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1 
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Abstract 

This  research  investigated  value  and  utility  functions  in  multiobjective  decision 
analysis  to  examine  the  relationship  between  them  in  a  military  decision  making  context. 
New  data  is  presented  and  data  from  an  earlier  study  is  also  analyzed  for  this  relationship. 
Further,  the  impact  of  these  differences  on  the  decision  model  was  examined  in  order  to 
improve  implementation  efficiency.  Specifically,  the  robustness  of  the  decision  model 
was  examined  with  respect  to  the  preference  functions  to  reduce  the  time  burden  imposed 
on  the  decision  maker. 

Data  for  decision  making  in  a  military  context  supports  the  distinction  between 
value  and  utility  functions.  Relationships  between  value  and  utility  functions  and  risk 
attitudes  were  found  to  be  complex.  Elicitation  error  was  significantly  smaller  than  the 
difference  between  value  and  utility  functions.  Risk  attitudes  were  generally  neither 
constant  across  the  domain  of  the  evaluation  measure  nor  consistent  between  evaluation 
measures.  An  improved  measure  of  differences  between  preference  functions,  the 
weighted  root  means  square,  is  introduced  and  a  goodness  of  fit  criterion  established.  An 
improved  measure  of  risk  attitudes  employing  utility  functions  is  developed. 

Response  Surface  Methodology  was  applied  to  improve  the  efficiency  of  decision 
analysis  utility  model  applications  through  establishing  the  robustness  of  decision  models 
to  the  preference  functions.  Which  value-utility  differences  are  significant  was 
detennined.  An  algorithm  was  developed  and  employs  this  information  to  provide  a 
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hybrid  value-utility  model  that  offers  increased  elicitation  efficiency  for  the  decision 


maker. 
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ROBUSTNESS  OF  MULTIPLE  OBJECTIVE  DECISION  ANALYSIS  PREFERENCE 


FUNCTIONS 


“When  all  available  knowledge  has  been  applied,  the  problem  is  reduced  to  one  of 
preference.”  (Matheson  and  Howard,  1968:  21) 


I.  Introduction 


Decision  Analysis  and  Operations  Research 

This  research  extends  decision  analysis  theory  and  knowledge.  All  Operations 
Research  (OR)  is  concerned  with  making  decisions.  The  introductory  remarks  in 
Introduction  to  Operations  Research  (Hillier  and  Lieberman,  2001 :  3)  opines  “to  be 
successful,  OR  must  . . .  provide  positive,  understandable  conclusions  to  the  decision 
maker(s).”  Decision  analysis  (DA)  provides  a  framework  for  examining  complex 
decisions  made  under  conditions  of  uncertainty,  particularly  for  single,  non-repeatable 
situations.  The  DA  literature  continues  to  expand,  testifying  to  its  success.  Besides 
extensions  to  theory,  DA  has  found  application  in  many  fields. 

Decision  analysis  provides  a  bridge  between  the  objectivist  and  subjectivist 
formulations  of  probability,  linking  the  decision  maker’s  intrinsic  attitudes  to  the  external 
environment  with  mathematical  tools.  In  a  sense,  DA  has  returned  to  the  roots  of 
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probability  theory,  to  the  work  of  Pascal  and  Bernoulli.  In  providing  this  confluence,  DA 
provides  a  solid  analytical  foundation  for  problems  of  increasing  complexity. 

Decision  makers  likely  will  face  increasingly  complex  situations  because  of 
increasingly  complex  technologies,  more  rapid  communications,  faster  business 
practices,  and  increasing  competition.  Frequently  decisions  involve  objectives  that  are  in 
conflict.  Acceptable  alternatives  are  challenging  to  develop.  Decisions  often  involve 
intangible  aspects  and  are  of  interest  to  multiple  groups  with  differing  views  and  may 
involve  multiple  decision  makers.  Often  decisions  involve  expertise  from  several 
unrelated  disciplines.  Uncertainty  also  has  a  significant  impact  on  decisions.  Important 
decisions  typically  involve  high  stakes,  are  complex  in  nature,  are  multi-disciplinary,  and 
need  to  be  defensible  (Keeney,  1982:  803  -  806).  All  these  aspects  of  decisions 
challenge  the  decision  maker,  and  the  operations  researcher.  Certainly  without 
quantitative  reasoning,  our  society  will  be  less  well  disposed. 


The  Purpose  of  Multiple  Objective  Decision  Analysis 

Keeney  (1982:  806)  defines  decision  analysis  as  “a  formalization  of  common 
sense  for  situations  which  are  too  complex  for  infonnal  use  of  common  sense.”  He  also 
offers  a  more  technical  definition:  “a  philosophy,  articulated  by  a  set  of  logical  axioms, 
and  a  methodology  and  collection  of  systematic  procedures,  based  on  those  axioms,  for 
responsibly  analyzing  the  complexities  inherent  in  decision  problems.”  The  term 
decision  analysis  is  sometimes  used  more  broadly  to  refer  to  the  analysis  of  any  decision 
(see,  for  example,  [Wallace,  2000]).  In  this  document,  the  term  decision  analysis  (DA) 
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refers  to  the  OR  methodologies  derived  from  this  axiomatic  basis,  those  that  produce  a 
preference  function. 

Multiple  Objective  Decision  Analysis  (MOD A)  is  employed  to  permit  the  decision 
maker  to  clarify  in  her  own  mind  which  alternative  she  should  select.  The  intent  of 
Multiple  Objective  Decision  Analysis  is  to  be  prescriptive,  to  provide  what  is  the  most 
prudent  course  available  in  a  bewildering  decision  context,  as  established  from 
examination  of  underlying  preference  components.  Multiple  objective  decision  analysis 
is  not  descriptive  in  that  it  does  not  attempt  to  replicate  how  an  individual  makes  a 
decision.  It  is  also  not  nonnative  in  that  it  accounts  for  deviation  from  preferences  that 
would  be  expected  of  one  who  was  strictly  a  formalist  in  application  of  mathematical 
principles.  (Keeney  and  Raiffa,  1993:  xv).  (A  less  fastidious  taxonomy  of  decision 
making  considers  the  categories  normative  and  prescriptive  to  be  identical.  For  example, 
see  Skinner  [1999:  13].) 

Operations  research  is  credited  as  being  the  first  organized  scientific  discipline 
devoted  to  decision-making.  Emerging  as  a  profession  during  the  Second  World  War, 
operations  research  analysts  played  important  roles  in  developing  effective  tactics  for  air 
and  sea  forces.  Postbellum,  those  skilled  in  OR  techniques  found  applications  in  the 
private  sector  and  operations  research  continued  to  develop.  The  problems  that  were  well 
suited  for  application  of  OR  tools  were  repetitive  in  nature.  Whether  searching  for 
enemy  aircraft  or  scheduling  production,  OR  techniques  were  found  to  be  useful. 
Operations  research  flourished  at  the  lower  and  middle  levels  of  management.  But 
executive  decisions  tend  to  be  unique,  not  repetitive,  involve  large  portions  of  an 
organization’s  resources,  characterized  by  high  degrees  of  uncertainty,  and  typically 
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involve  the  decision  maker’s  personal  risk  attitudes.  Operations  research  techniques  were 
found  lacking  for  application  in  the  boardroom.  (Matheson  and  Howard,  1968:  21  -  22) 
The  term  “decision  analysis”  is  attributed  to  Howard  (1963:  55).  That  DA  arose 
not  until  two  decades  after  the  genesis  of  OR  is  ascribed  to  three  influences.  First,  the 
development  of  electronic  computational  capability  fostered  the  ability  to  develop 
computationally  intensive  mathematical  models.  The  ubiquity  of  computers,  while 
assisting  greatly  in  ciphering,  had  a  negative  aspect  in  organizations.  Managers  were 
observed  to  say  that  computers  made  them  feel  as  if  they  had  less  control,  not  more,  over 
their  organizations.  Decision  analysis  provided  an  analytic  focus  on  managerial  issues, 
and  provided  for  employing  the  computer  as  an  appropriate  tool.  Finally,  private 
corporations  evolved  more  complex  managerial  structures  and  techniques.  Boards 
supplanted  individual  managers.  More  complex  financial  arrangements  increased 
auditing  activities.  These  trends  required  decisions  to  be  more  thoroughly  reasoned  and 
defensible.  (Matheson  and  Howard,  1968:  22  -  23) 

Decision  analysis  provides  a  number  of  advantages.  Besides  extending  analysis 
beyond  repetitive  decision  situations,  it  codifies  the  process  of  making  the  actual 
decision.  Prior  to  the  development  of  DA,  many  man-months  of  effort  were  devoted  to 
analysis  of  particular  problems.  This  analysis  may  have  been  development  and 
employment  of  a  simulation,  the  application  of  statistical  tools  to  historical  or 
experimental  data,  or  similar  efforts.  The  analytic  results  were  reduced  to  graphs,  tables, 
or  other  conveyances  and  presented  to  the  decision  maker.  The  decision  maker  then  was 
forced  to  absorb  the  infonnation,  process  it  in  some  internal,  usually  unspecified  manner, 
and  then  announce  his  decision.  Decision  analysis  permits  the  tools  of  operations 
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research  to  be  brought  to  bear  on  the  decision  itself  to  assist  the  decision  maker.  Also, 
the  decision  analytic  process  itself  has  inherent  advantages.  The  DA  process  forces  the 
decision  maker  to  think  hard  about  her  preferences.  While  this  requires  an  investment  of 
time,  generally  participants  indicate  that  the  exercise  was  beneficial  in  making  them 
devote  the  effort  to  gain  insight  into  the  problem.  (For  example,  see  Felli,  Kochevar,  and 
Gritton,  2000:  60  -  61).  Corner  and  Kirkwood  (1991:  206)  observe,  “Decision  analysis 
provides  tools  for  quantitatively  analyzing  decisions  with  uncertainty  and/or  multiple 
conflicting  objectives.  These  tools  are  especially  useful  when  there  is  limited  directly 
relevant  data  so  that  expert  judgment  plays  a  significant  role.”  Kirkwood  (1992b:  37) 
notes  that  DA’s  ability  to  perform  tradeoffs  among  multiple  objectives  makes  it  valuable 
for  government  decision  making.  Within  decision  analysis,  multiattribute  utility  theory  is 
considered  the  leading  methodology.  Keeney  and  Raiffa  (1993:  xi)  state,  “Decision 
analysis  is  widely  recognized  as  a  sound  prescriptive  theory.  When  a  decision  involves 
multiple  objectives  -  and  this  is  almost  always  the  case  with  important  problems  - 
multiattribute  utility  theory  forms  the  basic  foundation  for  applying  decision  analysis.” 

Decision  analysis  is  widely  applicable  to  military  decision  making.  Buede  and 
Bresnick  (1992)  show  that  decision  analysis  is  useful  in  the  materiel  acquisition  process. 
Vendor  engineers  viewed  value  models  distributed  with  the  request  for  proposal  for  the 
US  Marine  Corps  (USMC)  Mobile  Protected  Weapon  System  to  industry  as  helpful.  The 
USMC  found  that  all  ten  conceptual  responses  to  the  request  for  proposal  were 
“outstanding”  (115).  Later  work  for  the  USMC  involved  adapting  this  value  model  for 
off-the-shelf  procurement  of  the  Light  Armored  Vehicle.  A  utility  model  was  employed 
to  determine  the  best  air  defense  system  mix  for  a  US  Army  heavy  division.  The  authors 
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used  a  utility  model  to  assess  whether  control  of  the  Patriot  Air  Defense  System  should 
be  transferred  from  the  US  Army  to  the  US  Air  Force.  A  separate  study  group  conducted 
the  effort  in  parallel  with  a  simulation  approach.  The  two  efforts  were  in  concordance, 
the  DA  results  were  more  easily  interpreted  and  more  defensible,  and  the  simulation  was 
an  order  of  magnitude  more  expensive  and  slower  (121).  The  authors  include  a  list  of  26 
other  materiel  acquisition  projects  that  have  successfully  employed  decision  analysis. 
Other  notable  military  decision  analytic  efforts  are  SPACECAST  2020  (Burk  and  Parnell, 
1997)  and  AIR  FORCE  2025  (Jackson,  1996). 

Clemen  (2001:  6-8)  describes  the  decision  analysis  methodology  as  a  five-step 
process.  The  initial  step  is  to  identify  the  decision  situation  and  articulate  objectives.  The 
second  step  is  to  identify  alternatives.  Next  the  problem  is  decomposed  and  modeled. 
Three  aspects  are  modeled:  the  problem  structure,  the  uncertainty,  and  the  preferences. 
The  structure  of  the  problem  is  typically  modeled  using  decision  trees  and  influence 
diagrams.  Concepts  of  probability  theory  are  employed  to  capture  aleatory  aspects  of  the 
decision  problem.  Preferences  are  represented  with  value  or  utility  functions.  This  step, 
according  to  Keeney  (1982:  813),  is  unique  to  the  discipline  of  decision  analysis.  Next, 
the  modeling  effort  is  used  to  analyze  the  decision  and  the  best  alternative  is  chosen.  A 
sensitivity  analysis  is  typically  performed  as  the  last  step  to  detennine  the  robustness  of  a 
solution.  This  process  is  iterative,  and  repeated  until  an  adequate  solution  is  determined. 
Figure  1  shows  the  process  graphically.  (Alternate,  but  similar,  constructs  of  the  decision 
analysis  methodology  are  available  in  Skinner  [1999:  15]  and  Keeney  [1982:  807  -  820].) 
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Figure  1.  The  Decision  Analysis  Process  (after  Clemen,  2001:  6). 


Problem  Description 


Research  Questions.  The  research  primarily  examines  two  related  aspects  of 
decision  analysis.  Do  value  and  utility  functions  differ,  and,  if  they  differ,  when  are  the 
differences  important  and  how  should  this  be  considered  when  applying  decision 
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analysis,  especially  to  achieve  more  efficient  use  of  the  decision  maker’s  time? 

Regarding  the  first  point,  researchers  disagree  about  the  relationship,  if  any,  between 
these  two  preference  functions.  Empirical  data  consist  of  single  dimensional  decision 
problems  typically  presented  to  subjects  who  are  not  subject  matter  experts.  The  question 
exists  as  to  whether  professionals  exhibit  differences  in  value  and  utility  functions  for 
realistic  multidimensional  decision  problems.  If  differences  are  manifest,  is  a  functional 
relationship  discernable?  Additionally,  the  metrics  for  discerning  differences  between 
preference  functions  is  not  mature.  Only  a  general  metric,  root  mean  square  error,  has 
been  widely  accepted. 

Choice  of  either  form  of  preference  function  should,  except  in  certain  specific 
cases,  potentially  have  an  impact  on  identification  of  the  optimal  alternative.  The  second 
research  topic  addresses  the  robustness  of  decision  analysis  solutions  to  the  form  of  the 
preference  function.  When  one  accepts  that  there  is  a  distinction  between  value  and 
utility  functions,  when  is  it  necessary  to  acknowledge  that  distinction  in  application?  A 
more  pragmatic  question  is  germane,  how  may  a  value  function-based  model  be  adapted 
to  adequately  represent  the  behavior  of  the  corresponding  utility  function  model? 

Preference  Functions.  In  decision  analysis,  the  decision  maker’s  preferences 
under  certainty  are  represented  as  a  value  function  and  under  uncertainty  or  risk  as  a 
utility  function.  Value  functions  reflect  strictly  preferences  considered  in  a  deterministic 
sense,  while  utility  functions  reflect  those  preferences  as  well  as  risk  attitudes.  (In  this 
document,  all  references  to  value  functions  are  to  “measurable”  value  functions,  which 
provide  cardinality.  Non-measurable  value  functions  provide  only  ordinality  and  are  less 
useful  for  multiobjective  decision  analysis  models.)  However  this  interpretation  is  not 
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universal.  Opinions  in  the  literature  vary  widely.  Some  authors  have  used  the  terms 
interchangeably,  e.g.,  Anandalingam  (1987:  339).  Some  argue  that  they  are  identical.  For 
example,  see  Von  Winterfeldt  and  Edwards  (1986:  213).  Keeney  opines  that  when 
measurable  value  functions  and  utility  functions  for  a  subject  differ,  this  may  indicate  that 
the  elicitation  is  inaccurate  or  incomplete  (1992a:  187).  Others  believe  that  no  functional 
relationship  exists,  as  noted  by  Bouyssou  and  Vansnick  (1988:  1 10).  Finally,  another 
group  believes  that  a  functional  relationship  is  present,  e.g.,  Pennings  and  Smidts  (2000: 
1343). 

Value  functions  are  elicited  through  asking  questions  about  preferences  under 
conditions  of  certainty.  Typically  pairs  of  levels  of  an  attribute  are  compared  and 
adjusted  until  the  decision  maker  believes  that  quantitative  statements  about  his 
preferences  have  been  captured.  The  decision  maker  is  asked  to  compare  the  differences 
in  value  between  each  pair  of  attribute  levels.  Often  this  becomes  a  process  of  varying 
one  interval  until  the  value  increase  is  identical  to  that  of  the  other  pair.  Utility  functions, 
in  order  to  capture  the  uncertain  aspects,  are  elicited  employing  lottery-type  questions. 
The  decision  maker,  when  composing  an  answer,  must  consider  her  attitudes  about  the 
uncertainty  involved  as  well  as  preferences  for  various  levels  of  the  attribute  in  question. 
As  well  as  this  theoretical  difference,  empirical  data  tends  to  support  the  argument  that 
the  functions  are  distinct.  See  Pennings  and  Smidts  (2000:  1343)  for  a  summary. 

If  value  and  utility  functions  are  functionally  related,  work  by  Pratt  (1964:  122) 
provides  a  hypothesis  of  that  relationship  under  varying  risk  attitudes.  However 
empirical  data  has  failed  to  adequately  validate  these  relationships.  See  Keller  (1985b: 
481  -  482)  for  a  discussion.  The  situations  examined  by  Keller  are  restricted  to  the 
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univariate  case.  If  value  and  utility  are  separate  constructs,  the  value  function  may  be 
elicited  from  subject  matter  experts,  rather  than  the  decision  maker.  Then  the  utility 

function,  in  the  fonn  of  u  (v(jc))  may  be  elicited  from  the  decision  maker.  Such  a  process 

offers  the  potential  for  reducing  the  contact  time  required  with  the  decision  maker.  As 
decision  makers  often  face  great  time  demands,  such  a  reduction  makes  employment  of 
decision  analysis  more  practicable.  The  first  phase  of  the  research  examines  the 
relationship  of  value  and  utility  functions. 

The  initial  research  question  is  whether  w(x)  =  /(v(x)).  In  order  to  address  this 

question,  the  metric  for  determining  the  difference  between  preference  functions  will  be 
considered.  Are  current  metrics  acceptable?  A  third  question  becomes  what  criterion 
should  be  considered  representing  a  significant  difference  between  the  preference 
functions.  If  w  (x)  =  /(v(x)) ,  then  what  is  the  fonn  of  the  function/?  Does  the 

multiattribute  method  of  Kirkwood  provide  a  reasonable  approximation  of  single 
dimensional  utility  functions? 

Regardless  of  the  relationship  between  value  and  utility  functions,  other 
conjectures  and  observations  may  be  examined.  Do  individuals  maintain  consist  risk 
attitudes?  Does  elicitation  error  overshadow  the  elicited  differences  between  value  and 
utility  functions?  Does  simplifying  u  (x)  to  a  linear  function  in  x  provide  reasonable 
approximations? 

The  final  area  of  investigation  into  preference  functions  involves  the  decision 
attitudes  of  military  decision  makers.  Are  there  significant  learning  effects  present 
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during  elicitation  sessions?  Do  decision  attitudes  vary  with  experience,  rank,  education, 
or  military  occupational  specialty? 

Preference  Function  Robustness.  The  final  step  of  decision  analytical  effort  is  a 
sensitivity  analysis.  The  intent  is  to  see  how  sensitive  the  optimal  solution  is  to 
variability  inherent  in  the  problem.  A  sensitive  solution  may  indicate  that  additional 
analysis  is  desirable.  Alternatively,  it  may  indicate  that  the  best  alternative  is  not  the  one 
that  provides  the  highest  mathematical  expectation  for  the  objective  function.  Instead  an 
alternative  with  somewhat  lesser  desirability  under  nominal  conditions  may  retain  its 
value  relative  to  other  alternatives  better  under  varying  conditions.  Such  an  alternative  is 
considered  “robust.” 

Typically,  in  decision  analysis  this  is  done  in  univariate  fashion,  varying  a  single 
input  variable  and  observing  the  impact  on  the  overall  multivariate  value  or  utility 
function.  While  this  produces  valuable  insights,  it  does  not  permit  analysis  of 
interactions.  The  capability  to  do  pair-wise  examination  of  variables  has  been  added  to 
software  packages  in  the  last  few  years.  Until  recently,  the  state  of  the  art  was  to 
consider  only  three-way  sensitivity,  taking  three  variables  at  a  time.  See  Clemen  (2001 : 
183  -  184)  or  Borison  (2000:  530  -  531). 

Other  techniques  have  been  employed,  including  Dependent  Sensitivity  Analysis, 
Rank  Order  Stability  Analysis,  and  Response  Surface  Methodology  (RSM).  These 
techniques  are  more  powerful  than  simple  sensitivity  analysis  and  they  are  discussed  in 
Chapter  II.  To  the  author’s  knowledge,  no  sensitivity  analysis  been  employed  to  analyze 
preference  functions. 
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The  research  question  in  this  area  is  it  possible  to  conduct  sensitivity  analyses  on 
the  preference  function  itself?  If  this  is  tractable,  then  can  a  sensitivity  analysis  be 
employed  to  increase  the  operational  efficiency  of  decision  analysis  employment?  A 
hybrid  model,  composed  of  a  mixture  of  single  dimensional  value  and  utility  functions, 
may  be  a  useful  construct  that  reduces  the  time  required  of  a  decision  maker. 
Specifically,  is  it  possible  to  create  a  hybrid  model,  using  the  appropriate  value  model  as 
a  base,  that  approximates  the  behavior  of  the  utility  model?  If  so,  when  is  the  hybrid 
model  sufficiently  accurate  and  what  is  an  acceptable  criterion? 


Benefits  of  the  Research 

As  stated  above,  the  unique  aspect  of  decision  analysis  is  the  employment  of  the 
preference  function.  Yet  the  nature  of  this  function  is  in  dispute.  This  unsettled  central 
issue  leaves  decision  analysis  open  to  criticism.  Further  examination  of  this  issue  is 
clearly  important  for  the  continued  acceptance  of  decision  analysis.  This  research 
extends  previous  work  in  this  area  into  multivariate  cases. 

In  complex  problems  the  number  of  attributes  to  be  examined  is  large.  For 
example,  the  United  States  Air  Force  2025  Study,  which  was  commissioned  to  develop  a 
recommendation  for  allocation  of  that  service’s  funds  and  organizational  efforts  towards 
development  of  key  system  concepts  and  their  enabling  technologies,  had  134  attributes 
(Jackson,  1996;  and  Parnell,  Conley,  Jackson,  Lehmkuhl,  and  Andrew,  1998).  (An  effort 
for  the  National  Reconnaissance  Office  had  69  evaluation  measures  [Parnell,  2001].)  The 
attributes  ranged  from  operational  performance  issues  to  technical  feasibility  of  untested 
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technology.  Clearly  such  large  sets  of  attributes,  which  draw  on  expertise  from  multiple 
disciplines,  is  beyond  the  ability  of  a  single  individual  to  make  informed  preference 
statements.  Instead,  the  judgment  of  subject  matter  experts  must  be  coalesced  and 
presented  to  the  decision  maker.  Under  conditions  of  uncertainty,  common  for 
challenging  decision  situations,  utility  functions  must  be  elicited  personally  from  the 
decision  maker.  Such  an  approach  is  problematic,  as  for  large,  important  decisions;  the 
decision  maker  faces  many  competing  demands  upon  her  time.  The  Chief  of  Staff  of  the 
Air  Force,  the  senior  uniformed  officer  of  that  service,  for  example,  initiated  the  2025 
study.  A  methodology  that  permits  the  value  model  to  be  adapted  to  represent  the 
decision  maker’s  utility  model  with  the  least  decision  maker  interaction  would  be 
extremely  valuable  and  critical  for  consideration  of  decision  analysis  in  such  cases. 

The  research  extends  application  of  response  surface  methodology  within 
decision  analysis.  Current  analysis  of  uncertain  aspects  of  decision  analysis  problems 
lags  behind  developments  applied  in  other  research  areas.  Specifically,  response  surface 
methodology  will  be  employed  to  examine  the  sensitivity  of  the  decision  problem  to  the 
preference  functions  themselves.  This  is  a  novel  approach  in  sensitivity  analysis. 

Finally,  the  research  included  analysis  of  military  professionals’  preference  and 
risk  attitudes  for  a  realistic  tactical  problem.  While  certain  industries  and  public  sector 
groups  have  been  subject  to  intensive  decision  analytical  study,  no  similar  work  has  been 
performed  on  military  subjects. 
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Sequence  of  Presentation 


The  balance  of  the  dissertation  is  divided  into  five  chapters.  Chapter  II  will 
review  the  literature  regarding  value  and  utility  functions  in  decision  analysis  and 
response  surface  methodology.  Chapter  III  outlines  the  objectives  of  the  research, 
delimits  the  research  scope,  and  describes  the  methodology.  Chapter  IV  describes 
research  regarding  the  relationship  of  value  and  utility  functions.  Chapter  V  presents 
results  from  investigation  of  the  robustness  of  the  preference  function.  Chapter  VI  briefly 
summarizes  the  dissertation.  A  glossary  of  technical  terms  is  at  Appendix  A.  Details  of 
results  are  available  at  Appendices  B  and  C.  Appendix  D  contains  a  technical  report 
providing  the  decision  analysis  of  the  automatic  target  recognition  classification  systems 
problem.  This  work  is  extended  into  a  hybrid  value-utility  model  in  Chapter  V.  A 
bibliography  is  also  included. 
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II.  Literature  Review 


General 

As  discussed  in  Chapter  I,  decision  analysis  is  of  recent  development  in 
operations  research.  However,  the  concepts  involved  date  to  the  Enlightenment,  when 
mathematical  tools  were  brought  to  bear  on  decision-making.  That  work  provided  a 
foundation  upon  which  modern  economics  rests,  and  within  the  last  few  decades  has 
proven  to  be  the  inchoate  form  of  decision  analysis. 


History  of  Utility  and  Value  Theory 

Pascal.  Hacking  links  the  birth  of  scientific  analysis  of  decision  making  to 
Pascal,  and  his  well-known  application  of  probability  in  what  is  now  referred  to  as 
Pascal’s  Wager  (Hacking,  1975:  11).  Previous  work  with  probability  was  limited  to 
solving  puzzles  about  games  of  chance.  Here  Pascal  wrestles  with  the  concept  that  God 
either  is,  or  is  not.  If  God  exists,  one  should  lead  a  pious  life  leading  to  salvation.  If  not, 
one  may  enjoy  one’s  vices  without  regard  to  consequences  after  death.  However  the 
matter  is  unresolved  until  after  death,  and  the  choice  of  how  to  live  must  be  made  now. 
With  the  objective  probability  of  God’s  existence  unknown,  Pascal  focuses  on  the  idea  of 
the  differing  value  of  the  potential  outcomes,  extending  the  ability  to  analyze  uncertain 
situations  beyond  those  posed  about  gaming  tables.  He  is  responsible  for  recognizing 
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that  the  expected  value  of  an  uncertain  (discrete)  situation  with  n  outcomes  is  equal  to 
y^"xjpi  ,  where  xt  represents  the  potential  outcomes  and  pt  their  corresponding 

probabilities.  Pascal  vacillated  on  how  to  live  his  life,  sojourning  in  a  monastery,  and 
then  abandoning  his  refuge  to  the  exhilaration  of  the  casinos.  His  final  choice,  as 
salvation  was  preferable  to  damnation  regardless  of  the  probabilities  involved,  was  the 
decision  to  return  to  the  monastery  for  good.  (Bernstein,  1996:  69  -  71). 

Bernoulli.  The  development  of  the  concept  of  utility  theory  is  credited  to  Daniel 
Bernoulli.  In  a  1738  paper  published  by  the  Imperial  Academy  of  Sciences  in  St. 
Petersburg,  Bernoulli  states,  “the  value  of  an  item  must  not  be  based  on  its  price,  but 
rather  on  the  utility  that  it  yields.”  (Bernoulli,  1738/1954:  24).  He  notes  that  under 
conditions  of  uncertainty,  the  expected  value  of  an  outcome  is  determined  by  multiplying 
the  probabilities  and  the  respective  “gains.”  But  he  finds  this  inadequate  for  describing 
how  individuals  make  decisions.  He  proposed  that  an  individual  would  make  a  decision 
in  order  to  maximize  expected  utility,  rather  than  maximizing  the  expected  amount  of 
money  (ducats,  for  Bernoulli’s  paper)  or  some  other  objective  measure  of  worth.  The 
calculation  is  made  by  determining  the  mathematical  expectation  employing  utility  in 

place  of  the  units  of  currency,  employed  by  previous  researchers,  yielding  E  '«(* i)Pi  > 

where  u  is  the  utility  function.  He  also  asserted  that  for  a  given  decision  situation, 
different  individuals  would  assign  differing  utilities  to  various  possible  outcomes. 

Bernoulli  extended  his  analysis  by  suggesting  that  an  increase  in  utility  resulting 
from  a  gain  in  wealth  would  be  inversely  proportional  to  the  amount  of  previous 
possessions.  This  was  another  key  insight.  He  provided  a  graph  similar  to  Figure  2.  This 
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Wealth 


Figure  2.  Bernoulli's  Utility  Curve.  After  Bernoulli  (1738/1954). 


inverse  relationship  causes  the  concavity  shown  by  the  utility  function.  He  also 
demonstrated  that  two  individuals  with  identical  utility  functions,  who  engage  in  a  wager 
equally  likely  to  cause  either  to  pay  the  other  a  fixed  amount,  were  both  facing  a  loss  of 
expected  utility.  The  concavity  means  that  the  increase  in  utility  of  winning  is  less  than 
the  reduction  in  utility  of  losing  the  same  amount. 

Bernoulli  used  the  concept  of  utility  to  solve  a  question  originally  posed  by  his 
cousin,  Nicolas  Bernoulli  in  1713.  (With  Daniel  Bernoulli’s  solution,  this  problem  has 
become  known  as  the  St.  Petersburg  Paradox.)  Nicolas  had  developed  a  thought 
experiment  where  two  individuals  agree  to  a  game  between  them  with  no  house  take. 

The  first  player  will  flip  a  coin  repeatedly  until  the  result  is  tails.  Then  he  will  pay  the 
second  player  2"  ducats  (or  dollars)  where  n  is  the  number  of  coin  flips.  Before  the  game 
begins,  a  third  player  wishes  to  buy  the  right  to  play  from  the  second.  Nicolas  notes  that 
the  expected  value  of  the  game  in  a  monetary  sense  is  infinite,  as  the  payoff  increases 
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infinitely  and  the  probability  of  achieving  another  heads  never  reaches  zero.  But  most 
individuals  would  sell  the  game  for  some  amount  far  less  than  infinity,  say  between  5  and 
20  ducats  (or  dollars).  Daniel  Bernoulli’s  utility  provides  the  answer.  The  concavity  of 
the  utility  function  keeps  the  perceived  worth  of  the  gamble  far  from  infinity.  (Bernstein, 
1996:  99-  115). 

Using  a  geometrical  argument,  Daniel  Bernoulli  proposed  that  the  increase  in 
utility  for  an  individual  with  wealth  B  in  Figure  2  to  wealth  P  is  given  by 
PO  =  b  log  a  +  a/a  where  a  is  his  current  wealth  (equal  to  AB  ),  a  is  the  gain  (equal  to 
BP  ),  and  b  is  unique  to  the  individual.  Bernoulli  then  uses  this  relationship 
inappropriately  (by  misinterpreting  a  property  of  infinity  and  conveniently  dropping  the 
coefficient  b  ,  as  explained  by  Sommer,  [see  Bernoulli,  1738/1954])  to  show  that  for  a 
man  with  nothing  of  value  at  all,  the  expectation  of  the  game  would  be  two  ducats. 
Owning  ten  ducats  would  change  his  expectation  to  about  three  ducats.  He  argues  that 
very  large  initial  wealth  is  required  before  someone  would  offer  more  than  20  ducats. 
(Bernstein  [1996:  106]  observes  that  a  ducat  would  be  worth  about  40  dollars  in  today’s 
currency.)  Menger  showed  that  the  player’s  expectation  is  actually  equal  to 


b  log 


(a  +  1)^  (a  +  2)^  ■■■(a  +  2"  1  )• 


Moga  (Bernoulli,  1738/1954:  32). 


According  to  Bernstein,  Bernoulli  first  applied  measurement  to  something  that 
cannot  be  counted.  Bernstein  also  credits  him  with  “an  equally  profound  influence  on 
psychology  and  philosophy,  for  Bernoulli  set  the  standard  for  defining  human  rationality” 
(1996:  106).  Further,  Bernoulli  also  introduced  the  idea  of  human  capital,  that  one’s 
ability  to  produce  is  intrinsically  valuable. 
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The  concept  of  utility  was  recognized  as  valuable.  Adam  Smith’s  famous  passage 
from  The  Wealth  of  Nations  uses  the  concept.  Smith  opined  “The  word  value  has  two 
different  meanings,  and  sometime  expresses  the  utility  of  some  particular  object,  and 
sometimes  the  power  of  purchasing  other  goods  which  the  possession  of  that  object 
conveys.  The  one  may  be  called  ‘value  in  use’;  the  other,  ‘value  in  exchange.’”  (Smith, 
A.,  1775/1976:  44)  Here  we  see,  continued  from  Bernoulli,  imprecision  in  the 
employment  of  the  terms  value  and  utility  as  value  in  use  is  defined  as  the  utility  of  an 
object.  The  problem  persists  to  this  day. 

Bentham.  Jeremy  Bentham  widely  promulgated  the  concept  of  utility  in  England 
in  1789.  Bentham  applied  the  concepts  of  utility  to  all  aspects  of  society,  including 
assessing  the  appropriateness  of  penal  judgments.  Utility  theory,  although  donnant  for  a 
period  following  Bentham,  provided  the  foundation  for  later  development  of  economic 
theory,  particularly  the  Law  of  Supply  and  Demand,  and  remains  an  important  economic 
concept  today.  (Stigler,  1965:  89-92). 

Von  Neumann-Morgenstern.  The  developments  of  Bentham,  while  of  interest 
to  economists,  contain  little  of  direct  interest  to  the  development  of  decision  analysis. 
John  von  Neumann  provided  the  next  significant  development  in  utility  theory,  with 
respect  to  what  would  become  decision  analysis.  He  developed  game  theory.  In  a  simple 
game  two  opponents  each  make  a  binary  decision.  The  winner  is  determined  by  the 
results  of  the  pair  of  binary  decisions.  For  example,  in  a  game  where  the  players  secretly 
decide  to  show  (not  flip)  heads  or  tails  on  a  single  coin,  player  A  wins  if  both  show  the 
same  side.  Player  B  wins  if  one  shows  heads,  the  other  tails.  Von  Neumann  determined 
the  best  strategy  for  the  players  (which  is  a  random  display  of  heads  and  tails).  The  point 
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of  interest  is  that  the  laws  of  probability  do  not  dictate  the  optimum  strategy;  the  players 
themselves  do  so.  (Bernstein,  1996:  232  -  235). 

Later,  in  collaboration  with  Oskar  Morgenstern,  von  Neumann  published  the  book 
Theory  of  Games  and  Economic  Behavior.  (Von  Neumann  and  Morgenstern,  1953) 

They  proposed  an  axiomatic  basis  for  utility  theory.  The  axioms  provide  the  necessary 
and  sufficient  conditions  for  the  existence  of  a  utility  function.  Others  have  rewritten 
their  axioms  for  clarity.  For  their  description,  the  presentation  generally  follows  Biswas 
(1997:  17  -  18).  LetE  denote  the  outcome  set,  so  xeE  is  an  outcome.  Let 
X  =  (jct ,  x2 ,  •  •  • ,  xn )  be  the  vector  of  n  possible  discrete  realizations  from  the  outcome  set, 
each  with  a  respective  probability  of  occurrence,  pt .  Represent  a  lottery  of  these  n 
outcomes  with  L  =  (pxxx ,  p2x2  ,---,pnxn).  Preference  relationships  will  be  written  for 
xi,xj  el  as  xt  >-  x .  when  x;.  is  preferred  to  x . ,  and  x.  ~  x,  when  a  subject  is 
indifferent  between  x;.  and  x . .  Similarly,  to  represent  x;.  is  at  least  as  preferred  as  x , , 
we  write  x,.  V  x . .  Without  loss  of  generality,  we  will  assume  that  Xi^x,L"^;x„. 

Axiom  1  (Ordering).  Preferences  may  be  (totally)  ordered  between  any 
two  outcomes.  This  ordering  is  a  total  ordering.  So  for  any  (x,.,xy.)  either  x;.  V  xJ  or 

Xj  >Z  xt .  Further,  x,  V  xy.  and  xy.  V  xk  implies  xt  > 3  x}  > 3  xt . 

Axiom  2  (Reduction  of  Compound  Lotteries).  An  individual  is 
indifferent  between  a  simple  lottery  L,  (a  lottery  with  outcomes  that  are  fixed)  and  a 
compound  lottery  (a  lottery  with  one  or  more  outcomes  that  are  also  lotteries)  with  the 
same  outcomes  when  the  probabilities  of  the  simple  lottery  are  determined  in  accordance 
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with  standard  manipulation  of  probabilities.  If  a  lottery  is  defined  as 
L{m)  =  (p™xt ,  p2  x2 ,  •  •  • ,  p'"xn ),  where  the  superscripts  indicate  membership  in  a 

compound  lottery  m,  it  follows  that  (qxL^ ,q2l!-2\---,q0L^)~  (pxxx,p2x2,---,pnxn) 

where  pi  =  q,pf]  H - t- qo p\° 1  and  ~  represents  indifference. 

For  clarity,  this  axiom  will  be  detailed  for  the  case  of  two  lotteries  each  with 
possible  outcomes  xl  and  x2 .  These  lotteries  are  Z(1)  =  [p^x i ,  p^x 2 )  and 
Z<2)  =  (p[2)jc, , p^x2).  They  are  shown  graphically  in  Figure  3.  Then 
[qxL(x),  q2L(2\ ■  •  • ,  qoL(o])~  [pxx{ ,  p2x2 ,  •  •  • ,  pnxn )  becomes  {qxL(l\ q2L(2))~  (ptxt  ,p2x2) 
where  Pi  =  q.pf  +  •  •  •  +  qQp\o)  or  Pl  =  qlP[l)  +  q2p[2)  and  p2  =  q,p(2]  +  q2pf .  So  an 

individual  would  be  indifferent  between  the  choice  of  the  compound  lottery  or  the 
corresponding  outcomes  with  the  associated  reduced  probabilities, 

(qxL(l],q2L(2))~(p1xl,p2x2). 


Lottery  Z/'  ’  Lottery  L 


Figure  3.  Lotteries  L(1>  La> ,  and  L. 
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Axiom  3  (Continuity).  An  individual  will  be  indifferent  between  some 
xt  and  a  lottery  with  outcomes  x1  and  xn .  That  is,  there  exists  some  probability  qi  such 

that  x,  ~  (qix1 ,0x2 ,0x2 , •  •  • , (l  -qt )xn ) .  Define  this  lottery  Lx‘  =  (qix] ,(l  -q^x). 

Axiom  4  (Substitutability).  In  any  lottery,  Lx‘  may  be  substituted  for  x; . 
Axiom  5  (Transitive  Preference  over  Lotteries).  Preference  ordering  is 
transitive  over  lotteries.  That  is  L{l'>  A  [}  j)  and  ll ,l  A  [}k)  implies  A  L(  /l  A  qjk) . 

r\j  r\j  r  r\j 

Axiom  6  (Monotonicity).  An  individual  prefers  a  lottery 
L  =  ( pxx ,  (l  -  p)  x2 )  to  a  distinct  lottery  L’  =  ( p'x \ ,  (l  -  p')  x2  )  if  and  only  if  p>  p' . 

Similarly,  an  individual  is  indifferent  to  a  lottery  L  =  ( pxv  (l  -  p)x2  j  when  compared  to 

another  lottery  L'  =  (/i 'xl5(l -p ')x2)  ifandonlyif  p  =  p' . 

As  summarized  by  McCord  and  de  Neufville  (1983b:  279),  the  axioms  imply  the 
existence  of  a  cardinal  utility  function  (sometimes  referred  to  as  a  von  Neumann- 
Morgenstern  [vNM]  function)  that  orders  alternatives  independent  of  probability 
distributions.  Further,  this  function  may  be  used  to  rank  uncertain  alternatives  by  taking 
the  mathematical  expectation  of  the  vNM  utility.  That  is,  select  the  alternative  with  the 
highest  expected  utility  as  determined  by, 

u  =  YjWiE[ul(xl)]  (1) 

i=i 

for  each  alternative,  where  /,(/  =  1,2,  •••,/),  is  the  index  of  evaluation  measures,  vv;  are 
the  relative  weights,  x(.  is  the  level  of  the  ith  evaluation  measure,  and  ut  (x;)  are  the 
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corresponding  single  dimensional  utility  levels.  Hence  this  theory  is  referred  to  as 
expected  utility  theory. 


Criticisms  of  Expected  Utility 

Criticisms  of  Expected  Utility  (EU)  Theory  largely  are  rooted  in  the  observation 
that  it  is  possible  to  construct  situations  where  individuals,  who  subscribe  to  the  axioms, 
later  make  choices  that  violate  the  axiomatic  consequences.  The  most  famous  example  is 
that  of  Allais,  and  is  now  well  known  as  the  Allais  Paradox  (Allais,  1953:  503  -  505). 
Presented  two  sequential  lottery  choices,  most  subjects  make  alternative  selections  in 
each  that  are  mutually  incompatible  without  axiomatic  violations.  Consider  four 
lotteries.  The  notation  indicates  the  payoff  then,  parenthetically,  the  associated 
probability. 

Z,  : [$30,000  (0.33);  $25,000  (0.66);  $0  (0.0l)]  (2) 

Z2 :  [$25,000  (1)]  (3) 

Z3:  [$30,000  (0.33);  $0  (0.67)]  (4) 

Z4  : [$25,000  (0.34);  $0  (0.66)]  (5) 

Initially  the  subject  must  choose  between  Z,  and  L2 .  Most  subjects  choose  L2 .  Then  the 
subjects  who  chose  L2  generally  choose  Z3  when  presented  with  a  choice  between  Z3 
and  Z4  .  This  combination  of  choices  violates  expected  utility.  Choosing  L2  evinces 

0.33m  (30, 000)  +  0.66m  (25, 000)  +  0.0  1m  (0)  <  m  (25, 000)  (6) 
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which  may  be  rewritten 

0.33*/  (30, 000)  +  0.01m  (0)  <  0.34m  (25, 000)  (7) 

When  alternative  L3  is  preferred  in  the  second  decision  situation,  this  leads  to 

0.33m  (30, 000)  +  0.67m  (0)  >  0.34m  (25, 000)  +  0.66m  (0)  (8) 

which  simplifies  to 

0.33m  (30, 000)  +  0.0  1m  (0)  >  0.34m  (25, 000)  (9) 

Clearly  results  (7)  and  (9)  are  incompatible.  Either  the  subject  has  not  made 
choices  in  a  rational  manner,  or  expected  utility  theory  fails  to  adequately  represent 
rational  decisions.  Simon  (1972)  proposed  the  theory  of  bounded  rationality.  Perhaps 
the  ability  to  calculate  the  utility  differences  in  the  lotteries,  which  differ  only  slightly  in 
probabilities,  but  more  noticeably  in  payoffs,  is  beyond  the  ability  of  a  decision  maker. 

Tversky  and  Kahnemann  (1986)  proposed  that  the  presentation,  or  framing,  of  the 
decision  situation  was  important  and  responsible  for  the  Allais  paradox.  They  recast  the 
decisions  situations  as  compound  lotteries.  When  presented  in  this  manner,  the  majority 
of  subjects  made  choices  in  consonance  with  expected  utility.  The  explanation  offered  is 
that  the  subject’s  view  of  what  is  the  status  quo  is  important.  In  Figure  4  the  status  quo  is 
seen  as  a  $25,000  gain,  and  the  status  quo  in  Figure  5  is  a  gain  of  zero.  Decisions  are 
made  with  this  as  a  base  state.  Now  the  results  are  that  someone  who  chose  L2  usually 
chose  La  ,  consistent  with  utility  theory.  (Biswas,  1997:  4-8) 

Ellsberg  (1961:  653  -  655)  presented  another  famous  example.  Given  an  opaque 
container  in  which  are  30  marbles,  ten  of  red  and  the  balance  being  either  blue  or  yellow, 
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1/34 


$0 


$30,000 

$25,000 

$25,000 

$25,000 


Figure  4.  Allais  Paradox  Initial  Lottery,  as  a  Compound  Lottery 


Figure  5.  Allais  Paradox  Choice  Two,  as  a  Compound  Lottery. 


and  then  offered  two  sequential  bets  with  replacement:  (1)  Call  red  or  blue,  if  correct  win 
$5,000;  and  (2)  Call  red  or  blue,  if  wrong,  collect  $5,000;  Ellsberg  found  that  most 
subjects  choose  red  for  both  bets.  This  violates  expected  utility  theory  because  red  as  the 
choice  in  bet  (1)  evinces 

pr  ■  u  (5000)  +  (1  -  pr )  •  u  (0)  >  pb  •  u  (5000)  +  (l  -  pb  )  •  u  (0)  (10) 

where  the  subscripts  r  and  b  indicate  the  respective  marble  color.  The  expected  utility 
relationship  of  bet  (2)  is  captured  by 
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pr -u (0)  +  (l -p,.)-u (5000)  >  pb  -w(0)  +  (l- //,,)•  w(5000)  (11) 

Multiplying  both  sides  of  (1 1)  by  -1  produces 

-pr  •  u  (0)  —  (l  —  pr )  •  u  (5000)  <  -pb  ■  u  (0)  -  (l  -  ph )  •  u  (5000)  (12) 

which  may  be  rewritten 

—pr  ■  u  (0)  +  pr  •  //(5000)  <  —pb  ■  i/(0)  +  pb  ■  //(5000)  (13) 

Adding  n(0)  to  each  side  produces 

( 1  —  pr )  •  n ( 0 )  +  pr  -u (5000)  <  (l  -  Ph)-  m(0)  +  pb-u{5 000)  (14) 

Equation  (14)  may  be  rearranged  to  form 

pr  -«(5000)  +  (l- />,.)•  «(0)<  pb  -//(5000)  +  (l-pb)-//(0)  (15) 


Equations  (10)  and  (15)  contradict  each  other.  This  violates  expected  utility  theory. 
Ellsberg  advanced  the  argument  that  decision  makers  are  likely  to  place  more  confidence 
in  objective  rather  than  subjective  probabilities.  The  red  probabilities  are  objective  while 
those  for  blue  are  subjective,  leading  a  decision  maker  so  inclined  to  select  red. 

Outside  of  scientific  studies  (See  [Keller,  1992:  4-5]  for  additional  references), 
human  behavior  has  called  expected  utility  theory  into  question.  In  practice,  the  tenets  of 
expected  utility  theory  have  been  often  observed  to  be  violated,  in  the  opinion  of  critics, 
when  an  individual  demonstrates  both  risk  seeking  and  risk  averse  behavior. 
Documentation  exists  in  the  work  of  Appleby  and  Starmer  (1987),  Kahneman  and 
Tversky  (1979),  and  Schoemaker  (1980).  Prakash,  Chang,  Hamid,  and  Smyser  (1996: 
239  -  240)  provide  a  summary  of  common  observations.  Some  individuals  purchase 
insurance,  a  risk  averse  behavior,  yet  gamble,  which  is  risk  seeking.  Investors  will  hedge 
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in  some  investments  and  speculate  in  others.  Investors  buy  into  “long-shot  bets.” 
Managers  proceed  with  mergers  that  are  expected  to  provide  lower  profit  margins. 

One  explanation  is  that  an  individual  overestimates  the  probability  of  success.  An 
alternate  explanation  is  that  the  participation  in  the  gamble  itself  provides  utility  (Yaari, 
1965:  288).  A  latter  developed  explanation  is  that  risk  aversion  is  affected  by  the 
decision  context.  Recall  that  Bernoulli  stated  that  the  risk  aversion,  the  concavity  of  the 
utility  curve,  was  resultant  from  decreasing  marginal  utility  as  a  function  of  total  wealth. 
Viscusi  and  Evans  (1990:  353)  cite  research  where  one’s  health  determines  the  shape  of 
the  utility  function.  The  contextual  explanation  has  also  been  extended  to  include  the 
individual’s  self-assessment  of  one’s  own  competency. 

Obtaining  the  preference  function  in  expected  utility  is  demanding,  and  some 
authors  suggest  not  worth  the  effort.  “It’s  well  documented  that  decision  analysts  often 
run  into  trouble  getting  the  decision  maker’s  utilities,  which  turn  out  to  be  critical  to  the 
analysis,”  according  to  Levitt  (Samuelson,  2000:  25).  Edwards  and  Barron  (1994:  311) 
argue  that  elicitation  of  functions  is  onerous  enough  that  they  recommend  treating  utility 
as  a  linear  function  in  x.  Reneau  and  Blanthorne  (200 1 : 148)  observe  that  it  is  well 
established  in  the  literature  that  presentation  sequence  and  presence  of  extraneous 
information  influences  effectiveness  of  elicitation  sessions  without  causing  the  decision 
maker  to  be  aware  of  the  impact. 

Humphreys  (1982:  147  -  163)  provides  a  summary  of  several  criticisms  of 
Multiattribute  Preference  Theory.  He  states  that  often  risk  is  viewed  as  a  single  entity, 
generated  as  a  by-product,  which  may  be  assessed.  Then  activities  generating  risk  above 
a  threshold  are  avoided  or  mitigated.  He  states  that  this  has  been  named  by  others  the 
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“phlogiston  theory  of  risk”  (147).  Conrad  (1982)  states  that  DA  is  not  useful  for 
managing  risk  in  society  as  political  aspects  are  not  included  in  the  analysis. 

Most  work  in  the  expected  utility  area  has  centered  on  relaxing  axioms  in  an 
attempt  to  improve  the  descriptive  ability  (Kimbrough,  1994:  618).  This  has  led  to  the 
genesis  of  descriptive  theories  of  decision  making,  which  are  beyond  the  scope  of  this 
discussion.  Descriptive  theories  address  how  an  unassisted  decision  maker  makes  a 
choice.  They  do  not  attempt  to  indicate  which  choice  is,  in  fact,  the  best  alternative.  The 
reader  may  consult  Fishburn  (1989). 

Keller  (1992:  5)  visualizes  three  types  of  responses  to  the  observed  behavioral  EU 
violations.  First  is  the  acceptance  that  EU  is  normative  and  compartmentalizes  decision 
problems  into  those  where  EU  may  be  used  prescriptively  and  when  it  is  not  appropriate. 
The  second  are  attempts  to  improve  presentation  and  elicitation  tools  to  assist  the 
decision  maker  in  providing  coherent  responses.  Finally,  more  generalized  utility  models 
have  been  developed  that  have  descriptive  aspects  and  which,  perhaps  may  be  used 
prescriptively  under  appropriate  constraints. 

However,  despite  the  criticisms,  Kimbrough  (1994:  618)  remarked,  “For 
prescribing  behavior,  expected  utility  is  still  by  far  the  most  accepted  theory.”  This 
position  is  shared  by  Edwards  (1992:  xi),  who  points  out  experts  in  the  field  at  a  1992 
conference  “unanimously  endorsed  traditional  SEU  [subjective  expected  utility]  as  the 
appropriate  normative  model,  and  unanimously  agreed  that  people  don’t  act  as  the  model 
requires.”  Clearly,  expected  utility  is  the  preeminent  model  for  non-behavioral  aspects  of 
decision  making. 
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Measurable  value  functions 


Measurable  value  functions  were  developed  as  a  means  of  comparing  an 
individual’s  preferences  among  choices.  In  brief,  given  a  set  of  outcomes  {jCj ,  x2,  ■  ■  xn  } 

arranged  in  order  of  increasing  preference,  comparisons  can  be  made  between  pairs  of 
elements.  For  example,  someone  may  be  indifferent  between  the  respective  increase  in 
preference  between  certain  element  pairs,  (jct,jc;) .  This  is  read  as  the  increase 

in  preference  from  jc.  to  x ,  is  equal  to  the  increase  in  preference  from  xk  to  x, .  By 

establishing  such  relationships  and  normalizing  the  preferences  over  the  unit  interval  (or 
any  arbitrary  interval),  a  measurable  value  function  on  an  interval  scale  may  be 
determined.  Note  that  preference  statements  are  made  under  conditions  of  certainty,  and 
therefore  do  not  involve  the  subject’s  risk  attitudes.  Fishburn  (1967)  provides  a  summary 
and  classification  of  measurable  value  function  methods.  For  a  discussion  of 
measurement  theory  (scale  types),  see  Stevens  (1946),  Roberts  (1979)  or  Krantz  (1971). 
A  brief  summary  is  available  in  Fonnan  and  Gass  (2001:  470). 

Measurable  Value  Function  Axioms.  Farquhar  and  Keller  (1989:  205)  attribute 
the  “strength  of  preference”  concept  to  Pareto  and  Frisch.  Stigler  (1965)  provides  a 
history.  Also  known  as  “preference  intensity,”  this  is  a  quaternary  relationship 
comparing  preference  differences.  Krantz,  Luce,  Suppes,  and  Tversky  (1971)  provide  a 
set  of  axioms  that  establish  when  relationships  between  objects  may  be  defined  as  an 
algebraic  difference  structure.  (Other  sources  of  the  axioms,  or  alternative  formulations 
of  them,  are  [Fishburn,  1970],  [Suppes  and  Winet,  1955],  and  [Debrau,  i960].)  When 
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these  axioms  hold,  a  real-valued  function  exists  that  provides  cardinal  preference 
information  about  the  objects. 

Definition  (Algebraic  Difference  Structure).  The  notation  ab  represents  the 
increase  in  preference  resultant  from  receiving  object  b  in  lieu  of  object  a  .  Suppose  A 
is  a  nonempty  set  and  A  is  a  binary  relation  on  Ax  A.  The  pair  (A  x  A,  y'j  is  an  algebraic 
difference  structure  if  and  only  if  for  all  a,b,c,d,e,f  £  A  and  all  sequences 
ax ,  a2 ,  ■  ■  ■ ,  a(,  •  •  •  e  A  ,  the  following  axioms  hold  true: 

1 .  (AxA,^  is  of  weak  order.  A  relation,  i?,ona  set,  A  ,  is  of  weak  order  when 

it  is  transitive  and  strongly  complete.  Transitivity  exists  when  aRb  and  bRc  implies  aRc 
for  all  a,b,c  £  A .  A  relation  is  strongly  complete  on  a  set  when  aRb  or  bRa  holds  for 
all  a,b^A,  including  when  a  =  b  .  (Roberts,  1979:  20  and  29) 

2.  If  ab^cd  ,  then  dc^ba  . 

3.  If  ab  A  de  and  bc^ef ,  then  ac^df . 

4.  If  ab^cd^aa  ,  then  there  exist  e,f  <eA,  such  that  ae~  cd  ~  fb .  (The 
requirement  that  ab  A  cd  be  A  aa  ensures  that  ab  is  increasing.  This  is  required  so  that 
e,f  be  members  of  the  interval  \a,b] .  When  e,f  e  ab  then  it  must  be  true  that 

e,f  e  A.) 

5.  If  ax ,  a2 ,  •  •  • ,  at ,  •  •  •  is  a  strictly  bounded  standard  sequence  aMaj  ~  a2ax  for  every 
ai,aM  in  the  sequence,  not  a2ax  ~  a,  a, ;  and  there  exists  e,f  £  A  such  that  ef  >-  atax  >-  fe 
for  all  in  the  sequence),  then  it  is  finite.  (Krantz,  Luce,  Suppes,  and  Tversky,  1971 : 
151) 
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Krantz  et  al  show  that  if  (  A  x  A,£ 3)  is  an  algebraic  difference  structure,  then  there 


exists  a  real-valued  function  v  on  A  such  that,  for  all  a,b,c,d  e  A  ,  ab^cd  iff 
v(a)-v(b)  >  v(c)-v(J)  and  v  is  unique  up  to  a  linear  transformation.  Methods  of 

elicitation  of  measurable  value  functions  are  described  by  Fishburn  (1967  and  1976); 
Johnson  and  Huber  (1977);  Kneppereth,  Gustafson,  Leifer,  and  Johnson  (1974);  Dyer  and 
Sarin,  (1979  and  1982);  Farquhar  and  Keller,  (1989);  and  Camacho  (1980). 


Relationship  of  Value  and  Utility. 

Opinion  regarding  the  relationship  between  value  and  utility  functions  is  varied. 
Von  Winterfeldt  and  Edwards,  (1986:  213)  state  that  there  is  no  significant  distinction. 
They  argue  that  there  are  no  truly  riskless  situations,  so  decision  making  under  certainty 
is  never  actually  encountered.  Secondly,  they  argue  that  concavity  associated  with  risk 
aversion  may  be  explained  by  marginally  decreasing  return  of  value  or  through  a  regret 
expression.  They  also  argue  that  repetition  reduces  risk  aversion  and  most  choices  are 
truly  repetitive  over  a  lifetime.  Finally  they  opine  that  errors  associated  with  elicitation 
of  preference  data  exceed  the  distinctions  between  value  and  utility  functions.  McCord 
and  de  Neufville  (1983a)  provide  an  empirical  study  supporting  this  position. 

Other  authors  disagree.  Keeney,  in  his  book  Value -Focused  Thinking,  states  that 
value  functions  are  used  under  conditions  of  certainty,  and  utility  functions  under 
conditions  of  uncertainty.  He  further  indicates  that  his  value  functions  are  ordinal  only. 
Only  measurable  value  functions  and  utility  functions  provide  cardinality.  (1992:  132) 
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He  also  opines  that  when  measurable  value  functions  and  utility  functions  for  a  subject 
differ,  this  may  indicate  that  the  subject  has  a  hidden,  or  un-elicited,  objective  (1992: 

187).  Bouyssou  and  Vansnick  (1988:  110)  observe,  “Many  authors  have  argued  that  they 
see  no  reason  why  lottery  comparisons  should  coincide  with  preference  difference 
comparisons.”  Clemen  (1996:  553)  argues  that  for  decision  analysis  the  distinctions  are 
not  significant  for  applications.  Pennings  and  Smidts  (2000)  state  that  significant 
differences  between  utility  functions  and  value  functions  were  found  by  Krzysztofowicz 
(1983a  and  1983b),  Keller  (1985b),  Smidts  (1997),  and  Weber  and  Milliman  (1997). 
Their  “results  confirm  the  proposition  that  u(x)  and  v(x)  are  different  constructs.”  (1343) 
If  value  and  utility  functions  are  distinct,  it  is  intuitively  appealing  that  they  are 
related.  Both  measure  preference  in  a  given  decision  context.  Utility  has  the  additional 
consideration  of  probabilities  of  various  desired  and  undesired  outcomes,  as  well  as 
possibly  other  factors.  Bouyssou  and  Vansnick  (1988:  109)  state,  “That  some  idea  of 
strength  of  preference  interferes  with  risky  choices  is  hardly  disputable.”  They  continue 
(110),  “It  seems  many  features  do  influence  risky  choice  apart  from  strength  of 
preference.  Among  them,  we  feel  regret,  ...  disappointment,  ...  the  existence  of  a 
specific  utility  (or  disutility)  of  gambling,  the  misperception  of  probabilities  . . .  and  the 
avoidance  of  ruin  are  the  most  important  ones.”  They  state  (109)  “according  to  Bell 
[under  certain  conditions]  that  value  is  related  to  utility  either  by  an  affine  or  an 
exponential  transformation.”  They  also  offer  the  observation  that  just  because  value  and 
utility  are  both  on  an  interval  scale  that  does  not  mean  that  there  exists  a  function  that  is  a 
linear  transfonnation  from  one  to  the  other.  (107) 
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Von  Winterfeldt  and  Edwards  (1986:  213)  observe  that  as  of  that  writing,  little  of 
the  literature  addressed  the  relationship  between  value  functions  and  utility  functions. 

The  proposed  relationships  applied  only  under  constrained  conditions.  They  summarized 
the  concept  of  mapping  from  the  decision  space  into  a  preference  space  in  a  graph, 
adapted  in  Figure  6.  An  attribute  is  measured  on  some  scale,  transformed  into  value,  and 
then  the  corresponding  value  is  transformed  into  utility.  Short  cuts  are  also  possible, 
bypassing  intermediate  steps,  as  shown  in  Figure  7,  also  adapted  from  Von  Winterfeldt 
and  Edwards  (1986). 


Objects  — ►  Natural  or  Constructed  Scale  — ►  Value  Scale  ►  Utility  Scale 


Locations  of 

^  Driving  Distance 

Ratings  of  Relative 

Utilities  of  Values 

Apartments 

(in  miles)  from 

Value  of  Driving 

Driving  Distances 

4 

Office 

- ►  *Li) 

Distances 

— ►  v(d) 

— ►  u(v) 

100  100 


Figure  6.  Mapping  From  Decision  Space  Into  Preference  Space  (Von  Winterfeldt  and 

Edwards,  1986). 


Krzysztofowicz  (1983b)  makes  the  argument  that  u  =  w(v)  for  some 


transformation  w.  He  says  that  according  to  Pratt  (1964)  that  the  value  to  utility 
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Full  Decomposition 

Objects  -►  Natural  or  Constructed  Scale  -►Value  Sc  ale  “►Utility  Scale 
No  Value  Scale  (Keeney  and  Raiffa,  1976) 

Objects  -►Natural  or  Constructed  Scale - ►  Utility  Scale 

No  Natural  or  Constructed  Scales  (Bell  and  Raiffa,  1979) 

Objects  - ►  Value  Scale  -►Utility  Scale 

No  Natural  or  Constructed  Scale,  Value  Equals  Utility  (Edward,  1977) 

Objects - ►  Value  Scale  =  Utility  Scale 

No  Natural,  Constructed  or  Value  Scales  (Raiffa,  1968) 

Objects  - ►  Utility  Scale 

Figure  7.  Decision  Space  To  Utility  Space  Mapping  Sequences,  after  von  Winterfeldt 

And  Edwards  (1986). 


transformation  w  holds  under  the  condition  of  constant  relative  risk  aversion.  Using 
Pratt’s  measure  of  local  risk  aversion  c, 


c(x)  = 


d2u(x )  / du(x) 
dx2  /  dx 


(16) 


He  then  proposes  an  equivalent  value  function  measure  of  local  risk  aversion,  m, 


m(x)  = 


d2v{x)  J dv{x) 
dx2  /  dx 


(17) 


and  for  the  function  w,  utility  function  measure  of  local  risk  aversion,  n, 


d2w{v{xf)  I dw(v(x)) 
dv2(x)  /  dv(x) 


(18) 


He  then  offers  the  theorem  that  a  decision  maker  is  relatively  risk  averse  if  and  only  if 
n(x)  >  0  ,  relatively  risk  seeking  if  and  only  if  n(x)  <  0  ,  and  relatively  risk  neutral  if  and 
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only  if  n(x )  =  0  .  These  relationships  translate  to  concavity,  convexity,  and  linearity  of 
the  function  w. 

Krzysztofowicz  then  provides  a  theorem  for  the  transformation  w: 

i  -Ztv(x) 

u(x)  =  — 6  iff  n(x)  =  b  >  0  (1 9) 

1  -  e 

-bv(x)  _  i 

u{x)  =  — — - iff  n(x)  =  b  <  0  (20) 

e  -\ 

u(x)  =  v{x)  iff  n(x)  =  b  =  0  (21) 

Krzysztofowicz’  (1983b:  1 10  -  ill)  analysis  of  his  experimental  results,  as  well 
as  his  analysis  of  data  from  the  literature,  supports  that: 

•  Value  and  utility  functions  are  different  measures,  both  in  theory  and  as 
supported  in  use. 

•  The  classic  von  Neumann-Morgenstern  utility  function  (1953)  must  be 
reinterpreted  within  a  relative  risk  attitude  framework. 

•  Decision  makers  are  relatively  risk  constant  in  a  single  decision  context,  and 
that  the  function  w  holds. 

•  Relative  risk  attitude  is  not  constant  between  decisions. 

•  Group  decision  making  results  in  relative  risk  neutrality. 

Keller  (1985b)  examined  the  relationship  between  value  and  utility  functions  in 
29  subjects  for  one  of  four  decision  situations.  She  examined  the  fit  of  four  models  of  the 
relationship  between  value  and  utility:  linear,  v(x)  =  u(x) ,  exponential  (using 
Krzysztofowicz’  notation  w). 
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c(v(jt)) 


logarithmic, 


w(v(x))  = 


l-e 


1  -  e~c 


,c  ^  0 


,  ,  ,,  log(v(x)  +  c)  —  log  c  , 
w(v(x ))  =  V,  —  ,c  >  0 


log 


1  +  c 
V  c  ; 


(22) 


(23) 


and  power, 

w(v(x))  =  v(xy  (24) 

These  functions  provide  concave  or  convex  smooth  curves,  except  for  the 
logarithmic  function,  which  only  provides  concave  curves.  She  found  that  the  linear 
model  provided  the  best  fit  in  only  one  instance,  supporting  the  assertion  that  value  and 
utility  differ.  The  exponential  model  was  the  best  fit  in  nine  cases,  the  power  four, 
logarithmic  three,  and  for  twelve  subjects  no  model  produced  an  acceptable  fit  (root  mean 
square  error  less  than  0.05).  Her  conclusions,  the  last  two  of  which  she  cautions  are 
preliminary,  are: 

•  Generally,  v(x)  ^  u(x) . 

•  Risk  attitudes  varied  widely. 

•  Risk  attitudes  vary  for  a  subject  between  attributes. 

•  Constant  relative  risk  aversion  appears  to  not  be  appropriate  for  all 
subjects. 


The  final  point  differs  with  the  work  of  Bell  and  Raiffa  (1982)  in  that  they 
hypothesized  that  individuals  would  exhibit  constant  risk  aversion  in  all  attributes.  Keller 
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proposes  that  constant  relative  risk  aversion  does  not  always  produce  an  adequate  model 
of  decision  making.  She  also  observes  that  her  findings  are  in  disagreement  of 
Krzysztofowicz  (1983b),  who  found  the  exponential  family  to  provide  good  results. 


Methods  of  Comparing  Preference  Functions 

Several  methods  are  present  in  the  DA  literature.  Two  basic  forms  exist,  those 
that  evaluate  preference  functions  at  a  specific  point,  and  those  that  examine  preference 
functions  over  the  entire  domain.  Pratt’s  (1964)  (and  Arrow’s)  measure  of  local  risk 
aversion,  r(x)  =  -u”(x)/u'(x) ,  may  be  used  to  compare  two  functions  and  so  is  an 
example  of  the  former.  McNamee  and  Celona  (1987:  94)  suggests  that  an  exponential 
utility  function,  u{x)  =  a  -be  X,R  ,  be  fit  to  a  subject’s  preference  data  and  the  constant 
R>  0  is  defined  as  the  risk  tolerance.  Alternatively,  they  define  a  risk-aversion 
coefficient  as  1/R.  The  risk  tolerance  or  risk  aversion  coefficient  may  be  used  to  compare 
preference  functions  at  an  arbitrary  point  (i.e.,  outcome),  if  the  assumption  of  the 
exponential  function  is  acceptable. 

McCord,  and  de  Neufville  (1983)  compared  value  and  utility  functions  with 
“relative  difference.”  They  selected  an  ordinate  scale  position,  and  then  compared  the 
differences  between  the  corresponding  abscissa  values  for  two  functions  divided  by  the 
corresponding  abscissa  value  for  some  selected  reference  function.  Figure  8  shows  these 
values  for  two  functions  /  and  g  compared  to  a  reference  function  h  evaluated  at 
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Figure  8.  Measure  of  Relative  Difference  (McCord  and  de  Neufville,  1983:  292). 


/  (jCj )  =  g  (x2 )  =  h  (  jc3  )  =  A .  They  define  the  relative  difference  D  ( A )  f  as 


h-\A) 


(25) 


Fishbum  (1979)  proposed  RMSE,  which  pennits  evaluation  across  the  entire 
domain  rather  than  being  limited  to  a  specific  point.  Keller  (1985b)  employed  RMSE  in 
her  comparison  of  value  and  utility  functions  and  also  introduced  a  measure,  G,  for 
comparing  functions  to  the  linear  model  of  the  relationship  of  value  and  utility, 
u(x)  =  v(jc)  .  G  is  defined  as 

G  =  l_MSE(BestFtoi«gModel)  for  MSE  (Linear  Modd)  *  0  (2 

MSE  (LinearModel) 

When  MSE  for  the  linear  model  is  zero,  then  u  (x)  =  v(jc)  and  she  defines  G  to  be  zero. 

She  says  Fishburn  and  Kochenberger  (1979)  presented  a  similar  measure. 

Another  possible  discriminator  exists.  Kimbrough  and  Weber  (1994:  627) 
defined  a  measure  called  the  preference  area.  This  is  the  area  under  the  utility  curve, 
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with  the  domain  normalized  to  the  unit  interval.  A  risk  neutral  DM  would  have  a 


preference  area  equal  to  0.5.  A  risk  averse  DM  would  have  a  preference  area  greater  than 
0.5,  and  a  risk  seeking  decision  maker  would  have  a  preference  area  less  than  0.5.  When 
analyzing  data,  they  equated  risk  neutrality  to  the  interval  [0.45,  0.55].  The  preference 
areas  of  the  preference  functions  could  be  compared,  although  this  was  not  the  intent  of 
this  construct.  However  this  metric  would  not  discriminate  between  two  non-identical 
functions  that  had  the  same  area  under  the  curve. 


Decision  Analysis  Methodologies 

A  number  of  textbooks  are  dedicated  to  the  subject  of  decision  analysis  and  offer 
methodological  approaches.  A  selection  of  the  more  influential,  in  order  of  increasing 
age,  is  Clemen  and  Reilly  (2001),  Kirkwood  (1997b),  Keeney  and  Raiffa  (1993), 
Kleindorfer,  Kunreather,  and  Schoemaker  (1993),  Watson  and  Buede  (1987),  and  von 
Winterfeldt  and  Edwards  (1986).  Additionally,  the  work  by  Keeney  (1992a),  while  not  a 
textbook,  has  been  very  influential  in  decision  analysis. 

SMARTS.  A  number  of  methods  employing  the  underlying  tenets  of  value  and 
utility  for  decision  analysis  have  been  developed.  Within  the  decision  analysis 
community,  the  SMARTS  (Simple  Multiattribute  Rating  Technique  using  Swings) 
technique  is  well  known.  Originally  developed  by  Edwards  (1977)  it  has  undergone 
minor  modifications  over  time.  It  is  summarized  in  Edwards  and  Barron  (1994)  along 
with  some  simplifying  modifications  not  discussed  here.  The  steps  of  SMARTS,  in  the 
author’s  terminology,  are: 
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1.  Purpose  and  Decision  Makers.  Identify  the  purpose  of  the  decision  and  those 
who  will  make  it. 

2.  Value  Tree  (Hierarchy).  Elicit  a  structured  view  of  the  objectives  for  the 
decision. 

3.  Objects  of  Evaluation.  Define  the  alternatives  with  their  attribute  scoring 
scales  and  domain  limits. 

4.  Objects-by- Attributes  Matrix.  Create  a  matrix  comparing  alternatives  and 
scores. 

5.  Dominated  Options.  Remove  from  consideration  all  dominated  alternatives. 

6.  Single-Dimensional  Utilities.  Elicit,  and  then  replace  the  scores  in  the  matrix 
with  utility  levels. 

7.  Rank  Order  Attributes.  Order  in  declining  importance. 

8.  Quantify  Attribute  Weights.  Quantify  in  terms  the  most  important  attribute. 

9.  Decide.  Analyze  the  decision,  employing  an  overall  utility  model.  The  most 
common  is  the  additive  model. 

10.  Determine  the  utility  of  each  alternative  as  given  by 

<27> 

1=1 

where  h  (h  =  1,2,  •••,//)  is  an  index  of  alternatives,  i  (i  =  1, 2,  •  ■  • ,  I)  is  an 
index  of  attributes,  wt  are  the  weights,  xhi  is  the  z'th  evaluation  measure 
under  the  hth  alternative,  and  w,  (jcw)  are  the  corresponding  single 
dimensional  utility  levels. 
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For  our  purposes,  without  loss  of  generality,  the  preference  functions  will  range  in 
the  unit  interval  and  all  attribute  weights  sum  to  one.  That  is, 

K 

u(xi),v(xi)  e  [0,1]  =  [r  e  R :  0  <  r  <  1}  and  =  1  with  0  <  wk  <  1  for  all  k  .  Further,  in 

k= 1 

this  chapter,  increasing  levels  of  x  are  preferred.  Define  x0  as  the  least  preferred  level  of 
x  and  x,  as  the  most  preferred.  Then  we  have  u  (jc0  )  =  v  (jc0  )  =  0  and  u  (x*  )  =  v  (x* )  =  1 . 

These  conventions  are  well-known  in  the  decision  analysis  literature. 

Utility  Elicitation.  There  are  several  approaches  for  assessing  the  single 
dimensional  utility  functions.  The  Lock-Step  Procedure  selects  points  in  the  attribute 
domain  at  some  fixed  interval  (Kirkwood,  1997:  233).  At  each  point,  indifference  curves 
are  constructed  permitting  a  utility  function  to  be  established.  A  more  efficient,  and 
widely  used,  approach  is  the  Midvalue  Splitting  Technique  (Kirkwood,  1997b:  233  - 
235,  235  -  237).  The  domain  is  bisected,  and  the  utility  elicited  at  that  point.  The  two 
half-domains  are  similarly  bisected,  and  those  points  elicited.  This  procedure  may  be 
extended  until  the  desired  resolution  is  achieved. 

The  elicitation  of  the  utility  at  these  points  is  frequently  done  employing  one  of 
two  techniques,  the  certainty  equivalent  method  or  the  probability  equivalent  method.  In 
both  methods,  the  decision  maker  is  presented  a  decision  involving  a  choice  between 
uncertain  alternative  A  and  a  certain  alternative  B.  Alternative  A  has  two  possible 
outcomes.  One  occurs  with  probability  p  and  yields  outcome  x* ,  the  most  preferred 
level  of  x  .  The  other  occurs  with  probability  (l  -  p)  and  produces  x0 ,  the  least 
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preferred  level  of  x  .  Alternative  B  provides  a  sure  payoff  of  xB ,  where  x0  <  xH  <  x, . 


This  is  shown  in  Figure  9. 


Figure  9.  Decision  Diagram.  Squares  represent  decisions,  circles  uncertain  events, 
parentheticals  probabilities,  and  outcomes  are  expressed  as  the  variable  x. 


In  the  certainty  equivalent  method,  the  probability  p  is  equal  to  0.50,  so 
Alternative  A  is  a  50-50  bet  between  the  high  and  low  x  values  under  consideration.  The 
decision  maker  is  presented  various  values  of  xB  until  she  is  ambivalent  in  choosing 
between  the  two  alternatives.  By  the  expected  utility  axioms,  the  utilities  of  both 
alternatives  must  be  equal  at  ambivalence.  Further,  the  utility  of  x*  is  equal  to  one,  and 
the  utility  ofx0  is  equal  to  zero,  by  definition.  So 


u(xB)  =  (1 -  p)u(x  o)  +  (p)u(x„) 

(28) 

u(xB)  =  (.5)w(x0)  +  (.5)w(x*) 

(29) 

u(xb)  =  (*5)(0)  +  (.5)(1)  =  .5 

(30) 
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The  utility  may  next  be  determined  for  the  midpoints  of  the  subregions  [x0,x/;]  and 
[**>*•]• 


In  the  probability  equivalent  method,  xB  is  fixed  at  some  value  and  the 
probability  p  is  varied  until  the  decision  maker  is  ambivalent  between  the  alternatives. 
This  provides 


u(xB)  =  (  1  -  p)u(Xq)  +  (p)u(x,) 

(31) 

u(xB)  =  (1  -  p)(0)  +  Q0(1)  =  P 

(32) 

Since  p  is  known,  the  utility  of  xB  is  known.  The  utilities  of  subregions  are  then 
determined  in  similar  fashion  as  in  the  certainty  equivalent  method. 

One  the  utility  has  been  elicited  at  discrete  points  the  utility  function  is 
established.  A  typical  assumption,  that  of  constant  risk  aversion  pennits  the  fitting  of  an 
exponential  curve.  In  practice,  often  only  a  single  midvalue  point  is  used,  and  the  curve 
fitted.  (Kirkwood,  1997b:  235) 

Value  Elicitation.  Comparing  the  increase  in  preference  for  pairs  of  attribute 
levels  permits  value  function  elicitation.  Often  this  is  done  my  determining  a  midvalue 
level.  If  a  decision  maker  is  indifferent,  for  a  given  price  paid,  to  go  from  x0  to  xm  as 

from  xm  to  jc*  ,  then  xm  has  value  equal  to  0.5.  This  from 


=>  [V(Xm  )  “  V  (*0  )]  =  [V  (**  )  “  V  (X»>  )] 

(33) 

;U„)-0  =  l-v(xm) 

(34) 

2v(x,„)  =  1 

(35) 
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(36) 


Similar  comparisons  over  the  resultant  subregions  pennit  additional  value  point 
estimates. 

Weight  Elicitation.  Essentially,  this  is  typically  performed  by  starting  with  all 
attributes  at  their  lowest  levels,  then  “swinging”  one  attribute  at  a  time  from  its  lowest  to 
highest  levels.  Take  the  smallest  reported  increase  in  worth,  and  then  express  the  other 
increase  in  terms  of  the  smallest.  Normalize  the  weights  so  that  they  sum  to  one. 
(Kirkwood,  1997b:  68  -  70)  Clemen  (1996:  546  -  552)  also  provides  a  description  of 
this  technique  as  well  as  pricing  out  and  lottery  weight  techniques. 

Sensitivity  Analysis.  Sensitivity  analysis  has  a  role  in  all  decision  making,  not 
just  within  decision  analysis.  “In  solving  a  problem,  a  sensitivity  analysis  should  be 
made  in  order  to  detennine  the  variables  to  which  the  outcome  is  most  sensitive” 
(Operations  Analysis  Study  Group,  1977:  12).  A  typical  decision  analysis  effort  involves 
first  constructing  a  detenninistic  model  of  the  decision  opportunity.  Then  the  model  is 
examined  to  see  in  where  uncertainty  causes  important  effects.  Those  variables  where 
uncertainty  has  a  significant  impact  are  then  modeled  as  stochastic  while  the  remaining 
variables  are  handled  in  a  deterministic  manner.  The  detennination  of  which  variables 
should  become  stochastic  members  of  the  model  is  called  sensitivity  analysis  and  is  and  is 
ascribed  to  Howard  and  sometimes  referred  to  as  the  “Stanford  approach”  (Reilly,  T., 
2000:  55 1).  “The  motivation  behind  sensitivity  analysis  is  to  reduce  the  assessment 
burden”  (Reilly,  T.,  2000:  556).  Eschenbach  (1992:  41)  defines  sensitivity  analysis  as 
“examining  the  impact  of  reasonable  changes  in  base-case  assumptions.”  Skinner  (1999: 
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358)  links  the  employment  of  sensitivity  analysis  specifically  to  uncertain  aspects  of  a 
decision  problem,  rather  than  assumptions.  Eschenbach  provides  that  the  concern  is  for 
the  sensitivity  of  the  response  variable  to  differing  levels  of  independent  variables.  He 
lists  several  considerations  for  sensitivity  analyses:  (1)  establish  reasonable  bounds  for 
independent  variables,  (2)  establish  the  “unit  impact”  of  these  changes,  (3)  establish  the 
maximum  impact  of  the  independent  variables,  and  (4)  establish  the  change  required  in 
each  independent  variable  to  effect  alternative  choices. 

Stanford  Approach  to  Sensitivity  Analysis.  In  the  Stanford  approach, 
sensitivity  analysis  is  usually  conducted  by  singly  perturbing  each  variable  of  a 
deterministic  DA  model.  Reilly  (Reilly,  T.,  2000:  553  -  554)  provides  a  summary  of  this 
approach.  This  is  often  referred  to  as  “one-way  sensitivity  analysis”  (Clemen  and  Reilly, 
2001 :  179  -  183).  Variables  are  varied  singly,  and  the  effect  on  the  overall  preference 
model  is  observed.  When  the  objective  function  score  changes  considerably,  and 
especially  when  the  preferred  alternative  changes  within  the  range  of  change  of  the 
variable,  the  decision  problem  is  sensitive  to  this  variance.  Two-way  sensitivity  analysis 
is  also  done,  where  two  variables  are  varied  simultaneously. 

Reilly  (Reilly,  T.,  2000:  552  -  554)  summarizes  the  Stanford  approach  commonly 
as  modeling  the  variables  that  create  the  greatest  swing,  S0  ( V  )  ,  where  the  swing  is 
defined 

( ■ Xt )  =  max  {o  ( . Xu  ) ,  < o  ( . xib ) ,o  ( xih )  }  -  min  {o  (  xu  ) ,  o  (  xib ) ,  o  (  xih )  } 

where  X  =  (Aj ,  X2 , . . . ,  Xn  'f  is  the  column  vector  of  all  input  variables  (evaluation 
measures  and  relative  weights);  xa,xib,  and  xih  are  the  elicited  possible  low,  most 
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expected  (base),  and  possible  high  values  of  xt ;  and  o(x)  is  the  preference  (objective) 
function,  with  all  X k  it  Xt  fixed  at  xkb .  A  measure,  the  percent  variance  of  X. , 

PVE0  ( X,. ) ,  may  be  used  to  select  the  number  of  variables  to  represent  as  stochastic 
variables.  The  measure  is  defined  by 

PVE,(X,)=  fiA)]  (37) 

k= 1 

Reilly  states  that  it  is  common  to  see  use  of  PVEo  (A.)  lead  to  statements  similar  to 

“these  four  variables  capture  95  percent  of  the  variance  in  the  value  function.  (Reilly,  T., 
2000:  554).  He  points  out  that  such  an  interpretation  is  correct  only  if  the  variables  are 
independent  and  jointly  multivariate  normal. 

Rank  Order  Stability  Analysis.  Jolsh  and  Armstrong  (2000:  537  -  538) 
indicate  that  sensitivity  analysis  may  be  accomplished  through  a  technique  referred  to  as 
Rank  Order  Stability  Analysis  (ROSA)  and  attributed  to  Einarson,  Arikian,  and  Doyle 
(1995).  This  approach  is  basically  that  of  a  standard  sensitivity  analysis.  Instead  of 
varying  a  parameter  through  a  range  of  values  of  interest  and  observing  the  response  of 
the  objective  function  and  noting  the  change  in  optimum  alternative,  ROSA  merely  notes 
the  range  for  the  variable  that  retains  the  optimum  alternative.  This  is  restating  the 
standard  approach  results  but  presents  less  information.  However,  this  is  a  desirable 
technique  when  the  decision  maker  wishes  to  control  variables  to  maintain  the  optimum 
solution  rather  than  indicate  how  the  decision  responds  to  changes. 
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Jolsh  and  Annstrong  also  define  the  elasticity  of  a  variable,  E. ,  as  the  relative 


change  in  the  response  variable  to  a  relative  change  in  a  predictor  variable, 


where  v  is  the  response  variable  and  xi  an  independent  variable.  The  elasticity  provides 
information  analogous  to  the  response  surface  methodology  (RSM)  coefficient  for  xt . 

However  RSM  provides  statistical  tests  of  the  significance  of  the  tenn  and  also  is 
conducted  in  a  multivariate  framework  rather  than  the  ROSA  univariate  approach.  A 
multivariate  approach  pennits  the  consideration  of  variable  interactions. 

Dependent  Sensitivity  Analysis.  Reilly  (Reilly,  T.,  2000)  offers  a 
methodology,  dependent  sensitivity  analysis  (DSA),  for  sensitivity  analysis  where  the 
singular  value  decomposition  of  a  correlation  matrix  is  used  to  combine  the  independent 
variables  into  approximately  uncorrelated  synthetic  variables.  The  advantages  of  his 
method  are  that  independent  variable  dependencies  are  not  disallowed  and  the 
implementation  is  straightforward.  The  disadvantages  are  the  requirement  for 
additionally  eliciting  measures  of  dependence  relationships  -  Spearman’s  rank 
correlation,  the  requirement  to  assess  or  assume  the  standard  deviation  of  the  predictor 
variables,  the  loss  of  specificity  when  considering  independent  variables  grouped  into  the 
synthetic  variables,  and  the  additional  comprehension  burden  for  the  decision  maker  of 
the  concept  of  the  synthetic  variables  themselves. 

Reilly’s  DSA  uses  Spearman’s  rank  correlation  to  infer  dependency  among 
predictor  variables.  Asking  the  subject  directly  assesses  this  statistic,  although  he  advises 
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that  the  research  has  not  definitively  validated  this  approach  (2000:  555).  Using 
Speannan’s  rank  correlation  requires  that  monotonicity  be  assumed  and  that  the 
correlation  matrix  be  positive  definite.  The  DSA  approach  constructs  synthetic  variables 
that  are  affine  combinations  of  the  predictor  variables.  Let  the  matrix  of  rank  correlation 
estimates  Rs  =  ( rtj )  where  rtj  is  the  rank  correlation  for  Xt  and  X  . ,  i,j  =  \,2,...,n. 

Then  Rs  is  decomposed  through  singular  value  decomposition 

Rs  =  LALr  (39) 

where  LLr  =  LrL  =  I ,  the  columns  of  L  being  composed  of  the  orthonormal 
eigenvectors  have  Rs ,  and  A  is  a  diagonal  matrix  of  eigenvalues  of  Rs .  The  estimates  of 

the  variable  standard  deviations,  st  for  each  cr  associated  with  the  Xt ,  are  placed  into  a 
diagonal  matrix  S  .  The  synthetic  variables  are  determined  by 

Y  =  LrS^(X-M)  (40) 

where  X  =  (yXx,X2,...,Xnf  and  M  =  {xXb,x2b,...xnbf .  The  Yj are  approximately 
uncorrelated.  Rearranging  _  to  provide  sensitivity  analysis  produces  X  =  SLY  +  M .  For 

each  Xi  in  o(X)  the  substitution  becomes  X.  =  xib  +  si  (LitYt  +Li2Y2  H - b  Lin T )  for 

i  =  1, 2, . . . ,  n .  Reilly  states  that  this  substitution  is  readily  done  using  a  spreadsheet. 

Varying  a  synthetic  variable  Yt  causes  each  Xj  to  be  perturbed  by  the  addition  of 
LXSf .  The  direction  of  the  perturbation  is  detennined  by  L ..  and  the  magnitude  by  the 
product  of  L]t  and  st .  The  influence  of  the  L  jt  provide  that  correlated  X  f  vary  together. 
The  perturbation  of  each  Yt  should  cover  four  standard  deviations.  Because  the  synthetic 
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variable  variation  is  approximately  equal  to  the  eigenvalues,  Reilly  recommends  using 
varying  Yi  from  -2 to  2 .  The  L  matrix  is  also  useful  is  indicating  which  X, 

have  significant  influence.  Influential  X,  will  have  L -coefficients  “close  to”  |l|. 

Interpretation  of  the  results  is  similar  to  that  of  the  predictor  variables  except  that  the 
synthetic  variables  are  combinations  thereof.  For  example,  a  tornado  diagram  (see 
below)  may  be  constructed  of  the  synthetic  variables  just  as  would  be  done  with  the  basic 
predictor  variables. 

Response  Surface  Methodology.  Reilly  (Reilly,  T.,  2000:  570) 
comments  that  “No  systematic  approach  has  emerged  to  uncover  influential  groups  of 
variables  since  the  sheer  number  of  all  possible  multiple-way  sensitivity  analyses  make 
an  exhaustive  search  prohibitive.”  This  is  not  quite  correct,  although  the  information  was 
not  available  to  Reilly  when  he  prepared  his  initial  draft.  An  area  of  promise  is  the  work 
by  Bauer,  Parnell  and  Meyers  (1999:  162  -  180),  where  response  surface  methodology 
(RSM)  was  employed  as  a  demonstration  of  its  efficacy  for  decision  analysis  sensitivity 
analysis.  Response  surface  methodology  is  “a  collection  of  statistical  and  mathematical 
techniques  useful  for  developing,  improving,  and  optimizing  processes”  (Myers  and 
Montgomery,  1995:  1). 

Response  surface  methodology  provides  the  ability  to  model  some  system  that 
behaves  according  to  y  =  g  (jq,  x2, . . . ,  xn  )  +  s ,  where  g  is  unknown,  with 

y  =  f  (jCj ,  x2 , . . . ,  xn )  +  £  ,  where / is  determined  with  empirical  data.  Generally / is  either  a 

first  or  second-degree  polynomial,  and  forms  the  response  surface.  Figure  10  illustrates 
how  a  system  may  be  viewed  as  transforming  inputs  into  some  response  y  or 
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Figure  10.  Representation  of  a  System  Modeled  by  RSM. 

g  ,  •  •  • ,  )  =  y  ■  Response  surface  methodology  replaces  g  with  a  low  degree 

polynomial  f.  Response  surface  methodology  is  frequently  used  as  a  substitute  for  g  or  to 
gain  insight  into  the  important  aspects  of  g.  Response  surface  methodology  is  sometimes 
referred  to  as  regression  analysis  or  meta-modeling.  (Bauer,  2001 :  71) 

Response  surface  methodology  may  be  thought  of  a  polynomial  surface  over  a 
region  that  approximates  the  surface  of  g.  Each  alternative  in  a  decision  would  generally 
have  a  differing  surface  g.  The  independent  variables,  qi ,  have  upper  and  lower  bounds 

Lt  and  \Ji  respectively,  when  considered  across  all  alternatives.  Generally  in  RSM  the 

f  P  -P  ^ 

independent  variables  are  coded  xt  =  2  — — —  for  i  =  1,2,..., n  where  qi{]  is  the 

\U.  -L) 

midpoint  between  L  and  Ui .  This  maps  L  and  U t  to  -1  and  +1,  respectively,  offering 

the  advantage  of  working  with  unitless  variables  and  facilitating  experimental  design. 
(Bauer,  Parnell,  Meyers,  1999:  164) 

Response  surface  methodology  produces  a  metamodel  which  models  the  response 
of  the  decision  analysis  model  as  the  input  is  varied.  The  response  is  the  multiattribute 
preference  function,  and  the  input  the  decision  parameters.  The  hypothesized  model 
describes  the  response 
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y  =  Pa  +Z/U  +Z^¥i  +  XAxA  £'  ~  ^(°^2)  (41) 

/=1  /<  j  i= 1 

where  the  /?  are  unknown  coefficients.  This  approach  is  analogous  to  a  Taylor’s  series 
expansion  of  degree  2  for  the  true  functional  relationship.  These  coefficients  provide 
insight  as  to  contribution  of  each  variable,  as  well  as  the  strength  of  variable  interactions. 
Equation  (41)  may  be  simplified  by  omitting  the  xf  term  and  the  x(x;  interaction  term 

may  also  to  neglected.  As  simplified,  the  expression  is  easier  to  solve  and  so  is  often 
used  as  a  screening  device  early  in  the  RSM  process.  Employing  the  complete  Equation 
(41)  near  the  subregion  of  concern  provides  a  better  fit  of  the  approximated  surface.  The 
least  squares  estimated  coefficients  of  the  hypothesized  model 

y  =  b o  +  X  bixi  +  X  baxix]  +  X  b«x f  (42) 

(=1  i<  j  i= 1 

is,  in  vector  notation, 

b  =  (X'X)  1  X'y  (43) 

The  fitted  surface  is  provided  by 

y  =  Xb  (44) 

and  residual  error  is 

e  =  y-y  (45) 

The  coefficient  provides  a  ready  indicator  of  the  contribution  for  the  associated  variable 
and  statistical  tests  provide  a  measure  of  significance.  (Myers  and  Montgomery,  1995: 
16-21) 
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Bauer,  Parnell,  and  Meyers  present  Figure  1 1,  a  graphical  representation  of 
iterative  application  of  RSM  to  a  DA  problem  (1999:  165  -  166).  To  reduce  the  analytic 
burden,  variables  that  are  believed  to  behave  similarly  may  be  grouped  and  a  RSM 
screening  conducted.  Those  groups  that  have  significant  effects  are  then  screened  as 
individual  factors.  Significant  variables  are  then  modeled  with  a  first  order  equation  and 
the  fit  is  tested.  If  the  fit  is  insufficient,  a  second  order  model  is  employed. 


Group  1 

Group  2  { 

Group  3  — 


Group  1 


{ 


Group  3  — 


Figure  11.  RSM  Paradigm  (After  Bauer,  Parnell,  and  Meyers,  [1999:  166]). 


Group  Screening.  As  Bauer,  Parnell,  and  Meyers  (1999)  employ  group 


screening,  it  deserves  additional  discussion  beyond  a  review  of  RSM.  Meyers  and 


Montgomery  (1995)  fail  to  mention  group  screening  in  their  RSM  text.  A  screening 


design  in  experimentation  is  a  selection  of  experimental  inputs  that  permit  establishment 
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of  which  factors  are  significant  while  minimizing  the  number  of  experimental  iterations 
(usually  referred  to  as  runs  in  modeling)  and  so  reducing  associated  costs.  Large 
numbers  of  factors  force  a  prohibitively  large  number  of  runs  or,  when  the  number  of 
runs  is  constrained,  causes  factors  to  be  confounded  and  effects  masked.  Madu  and  Kuei 
(1992:  96)  caution  that  screening  large  numbers  of  variables  is  to  be  avoided.  Group 
screening  reduces  the  number  of  factors  considered  by  consolidating  them  into  groups. 
These  groups  are  then  manipulated  simultaneously  so  the  experimental  design  has  been 
reduced  from  the  total  number  of  factors  to  the  number  of  groups. 

O’Geran  and  Wynn  (1992)  provide  a  history  of  the  group  screening  methodology. 
The  concept  was  first  employed  in  1943  to  quickly  screen  human  blood.  Samples  from 
donated  units  were  combined  and  tested.  With  no  positive  results  in  pathological  tests 
the  units  were  quickly  cleared  for  use.  When  test  results  indicated  pathogens  were 
present,  individual  units  were  screened.  This  concept  was  applied  to  the  problem  of 
detection  of  significant  factors  by  Watson  (1961).  Kleijnen  (1975:  488)  indicates  that 
Patel  and  Li  generalized  this  two-step  process  into  n-steps  independently  in  1962. 

In  the  procedure,  stated  simply,  variables  are  first  aggregated  into  groups. 

Various  authors  have  developed  recommended  differing  grouping  assumptions  and 
strategies.  Then  an  acceptable  experimental  design  based  on  the  groups  is  selected. 

When  the  runs  are  complete,  all  variables  contained  in  the  groups  found  not  to  be 
significant  are  assumed  to  be  non-significant.  Significant  groups  are  decomposed  and 
new  groups  formed,  if  required,  and  the  process  repeated.  Group  screening  has  not  been 
found  in  the  DA  literature  beyond  the  work  by  Bauer,  Parnell,  and  Meyers  (1999).  Their 
efforts  are  shown  graphically  in  Figure  11. 
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While  standard  decision  analysis  sensitivity  analysis  identities  which  variables, 
without  interaction,  cause  the  solution  to  be  sensitive,  and  DSA  addresses  interactions 
employing  synthetic  variables,  RSM  provides  a  quantitative  estimate  of  the  impact  of 
each  variable  and  interactions. 

Communicating  Sensitivity  Analysis  Results.  Single-dimensional 
sensitivity  analysis  results  are  usually  presented  as  a  rainbow  diagram.  This  is  a 
Cartesian  plot  of  the  objective  function  as  a  function  of  one  of  the  predictor  variables,  the 
others  held  constant  at  their  expected  value  levels.  The  area  under  the  curve  is  normally 
color-coded  to  indicate  which  alternative  is  the  most  optimal  and  this  is  responsible  for 
the  name.  An  example  rainbow  diagram  is  shown  in  Figure  12.  A  tornado  diagram  is 
used  to  show  the  effect  of  singly  varying  multiple  predictor  variables.  An  example  is 
provided  in  Figure  13.  The  change  in  the  dependent  variable  as  each  independent 
variable  is  varied  is  indicated  by  horizontal  bars.  Bar  endpoints  are  positioned  on  a  scale 
of  the  independent  variable,  so  bar  length  is  a  measure  of  the  sensitivity  of  the  dependent 
variable  to  the  corresponding  independent  variable.  Bar  color  is  changed  to  indicate 
differing  alternatives  have  achieved  optimality.  The  bars  are  arrayed  in  descending  order, 
creating  a  graphic  evocative  of  a  tornado  moniker.  This  graphic  device  can  accommodate 
a  large  number  of  independent  variables  and  quickly  shows  their  relative  importance. 

Eschenbach  (1992:  42  -  43)  promulgates  the  spiderplot  as  an  adjunct 
graphical  device  to  display  sensitivity  analysis  results.  The  spiderplot  is  a  Cartesian  plot 
of  response  the  dependent  variable  to  percentage  change  in  independent  variables  where 
each  independent  variable  is  represented  by  a  curve.  The  curves  intersect  at  the  base 
point  (unperturbed  independent  variables)  creating  a  spider  web-like  appearance.  The 
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Figure  12.  Sample  Rainbow  Diagram. 
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Figure  13.  Sample  Tornado  Diagram. 


advantages  of  a  spiderplot  as  compared  to  a  tornado  diagram  are  that  the  former  shows 
the  respective  selected  independent  variable  limits,  the  unit  effect  of  changes  in  the 
independent  variables,  and  the  level  of  maximum  impact  of  the  independent  variables. 
The  disadvantages  are  the  increased  sophistication  required  to  understand  the  diagram, 
tendency  for  the  diagram  to  be  incorrectly  prepared,  and  the  number  of  independent 
variables  is  restricted  to  a  small  number  for  clarity.  Eschenbach  does  not  discuss 
presenting  the  effects  of  interactions.  An  example  of  a  spiderplot  is  shown  in  Figure  14. 
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Figure  14.  Sample  Spider  Plot.  After  Eschenbach  (1992:  43). 


Clemen  and  Reilly  refer  to  the  problem  of  presenting  results  of  sensitivity  analysis 
of  two  or  three  stochastic  variables  (2001 :  183  -  192).  They  observe  that  this  is  a 
“difficult  problem”  (200 1 :  183).  Their  solution  is  the  sensitivity  graph.  For  two 
variables,  a  Cartesian  plot  is  prepared  with  each  axis  representing  various  levels.  The 
plot  area  is  partitioned  into  regions  by  optimal  solution,  generating  a  plot  analogous  to  a 
phase  diagram  in  chemistry.  The  plot  may  be  extended  into  three  dimensions  to  present  a 
response  surface.  This  represents  the  state  of  the  art  in  DA  software.  For  example, 

DPL®  provides  a  “two-way  rainbow  diagram”  that  examines  changes  in  two  variables 
simultaneously  (Borison,  2000:  530  -  539)  as  illustrated  in  Figure  15.  For  three 
variables,  if  these  may  be  expressed  as  a  mixture,  a  ternary  (or  trilinear)  chart  may  be 
used  (Schmid,  1954:  170  -  171).  At  least  one  software  package,  Data®,  provides  for 
“three-way  sensitivity  analysis”  in  the  form  of  a  “two-way”  animated  graph  (Data  3.0, 
1996:  219  -  220).  The  user  interactively  varies  the  third  variable  and  observes  the  affects 
on  the  two-dimensional  plot. 
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Figure  15.  Sample  Two-Way  Rainbow  Diagram. 


Sensitivity  Analysis  of  Preference  Functions.  Sensitivity  analysis  in  the 
decision  analysis  literature  excludes  discussion  of  preference  functions.  The  literature 
does  not  address  the  issue,  although  it  does  provide  evidence  that  the  issue  is  of  import. 
Ghosh  and  Ray  (1997)  found  that  decision  maker’s  choices  are  both  a  function  of  risk 
attitudes  and  ambiguity  tolerance.  Additionally,  their  results  suggest  that  the  decision 
maker  often  interprets  ambiguity  in  a  decision  situation  as  risk.  This  indicates  that  the 
preference  function  likely  involves  some  inherent  uncertainty. 

Besides  the  uncertainty  present  within  the  decision  maker,  elicitation  sessions  are 
vulnerable  to  human  perception  frailties.  An  example  of  this  includes  framing,  where  the 
same  situation  presented  in  differing  manners  elicits  consistently  differing  preferences. 
Anchoring  is  the  phenomenon  in  which  an  individual  presented  with  some  estimate  of  a 
parameter  of  interest  will  often  provide  biased  elicitation,  assessing  higher  probabilities 
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against  the  estimate  while  significantly  underestimating  probabilities  of  severe  events. 
Further,  it  is  well  documented  that  individuals  will  demonstrate  more  confidence  in 
objective  probabilities  than  subjective  probabilities  without  discernable  justification.  See 
Kirkwood  (1997b:  1 10  -  127;  299  -  320)  for  discussion  of  these  issues. 


Recommended  Future  Research 

Keller  examined  the  relationship  between  value  and  utility  functions  in  29 
subjects  for  one  of  four  decision  situations.  She  recommended  that  future  research 
examine  risk  attitudes  and  relative  risk  attitudes  across  multiple  attributes  within  subjects, 
and  similarly  within  attributes  across  subjects.  She  also  recommended  that  future  work 
examine  Bell  and  Raiffa’s  (1982:  345)  conjecture  that  constant  relative  risk  aversion 
should  be  observed.  Finally,  she  recommended  repetitive  elicitations  in  order  to  assess 
elicitation  error.  Keller  has  indicated  that  she  believes  these  remain  “a  fruitful  area  for 
further  study”  (2000:  1). 

Keller  (1989a),  Brown  (1992),  and  Howard  (1992)  have  all  called  for  research  on 
prescriptive  decision  analysis  methods.  While  differing  in  recommended  approaches  and 
definition  of  the  term,  they  all  use  the  term  prescriptive  to  mean  procedures  that  reduce 
the  negative  aspects  of  employment  of  decision  analyses,  while  retaining  the  value  DA 
provides.  In  general,  they  see  reducing  the  burden  placed  on  the  decision  maker  as 
fostering  acceptance  of  the  employment  of  DA. 
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III.  Research  Overview 


The  research  had  two  major  objectives,  both  pertaining  to  the  relationship  of 
value  and  utility  preference  functions  in  decision  analysis.  This  chapter  will  restate  the 
objectives,  summarize  background  material,  provide  more  detail,  delineate  the  scope  of 
the  research,  and  provide  a  summary  of  the  methodology.  Detailed  explanation  of  the 
methodology  accompanies  the  presentation  of  the  analysis  and  results  in  Chapters  IV  and 
V.  Results  are  summarized  in  Chapter  VI. 


Objectives 

The  major  objectives  were  decomposed  into  supporting  subobjectives. 
Additionally  the  first  major  objective,  examining  the  relationship  of  value  and  utility 
functions,  supports  the  second.  The  second  objective  is  predicated  on  the  observation 
that  value  and  utility  functions  differ. 

Objective  1.  Examine  Relationship  Between  Single  Dimensional  Value  And 
Utility  Functions.  Examine  the  single  dimensional  preference  (value  and  utility) 
functions  of  a  group  of  military  subjects  within  the  context  of  a  multiattribute  decision 
situation. 

Objective  1.1.  Determine  Functional  Relationship  of  Single 
Dimensional  Value  and  Utility  Functions.  Examine  whether  single  dimensional  value 
and  utility  functions  have  any  consistent  functional  relationship,  including  whether  they 
are  equivalent.  Consider  the  exponential,  logarithmic,  power,  linear,  and  sigmoid 
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functions  as  candidates  for  this  relationship.  In  particular,  scrutinize  the  linear  case, 
which  corresponds  to  the  situation  where  the  value  function  equals  the  utility  function. 

Objective  1.2.  Examine  Characteristics  And  Performance  Of  Various 
Common  Decision  Analysis  Elicitation  Methodologies.  Examine  whether  single 
dimensional  utility  functions  elicited  using  the  certainty  equivalent  and  probability 
equivalent  methods  provide  significantly  differing  constructs  of  a  subject’s  preferences. 
Examine  whether  the  order  of  elicitation  affects  information  quality.  Establish  whether 
the  multiattribute  utility  approach  provides  an  acceptable  substitute  for  single 
dimensional  utility  models.  When  a  subject  fails  to  reach  ambivalence  under  the 
multiattribute  utility  approach,  examine  whether  bounding  of  the  multiattribute  utility 
function  provides  a  sufficient  answer. 

Objective  1.3.  Improve  The  Ability  to  Compare  Between  Preference 
Functions.  Develop  an  improved  method  for  discriminating  between  preference 
functions.  Ascertain  whether  the  elicitation  error  exceeds  the  differences  between 
preference  functions.  Establish  acceptance  criteria  for  fits  of  such  models  to  empirical 
data. 

Objective  1.4.  Examine  The  Subject’s  Risk  Attitudes.  Examine  the 
subject’s  risk  attitudes  for  constant  relative  risk  aversion.  Examine  the  concept  of  relative 
consistent  risk  attitude,  where  a  subject  maintains  consist  risk  attitudes  across  evaluation 
measures. 

Objective  1.5.  Improve  The  Metric  For  Assessing  Risk  Aversion. 

Develop  a  new  measure  of  risk  aversion. 


60 


Objective  2.  Establish  Methods  Of  Increased  Efficiency  Of  Decision 


Analysis  Elicitation  By  Considering  Robustness  Of  Employing  Single  Dimensional 
Value  Functions  As  Surrogates  For  Utility  Functions.  Examine  the  perfonnance  of  a 
hybrid  value-utility  model  as  a  more  efficient,  with  respect  to  elicitation,  DA  model  than 
a  standard  utility  model. 

Objective  2.1.  Establish  When  Value-Utility  Differences  Are 
Significant.  Employ  response  surface  methodology  and  principles  of  design  of 
experiments  techniques  to  assess  when  the  form  of  the  preference  function  itself  is 
significant  with  respect  to  the  decision  analysis  model. 

Objective  2.2.  Develop  an  Algorithm  for  Employment  of  a  Hybrid 
Value-Utility  Model.  A  hybrid  DA  model  adapts  an  existing  value  model  to  adequately 
represent  the  corresponding  utility  model.  The  algorithm  will  prioritize  the  iterative 
procedure  as  well  as  specify  stopping  criteria. 


Background 

In  decision  analysis,  a  model  of  the  decision  problem  is  manipulated  to  gain 
insight.  The  model  must  satisfactorily  mimic  the  decision  to  be  studied.  The  model 
estimates  will  deviate  from  some  central  value  due  to  external  and  internal  variation. 

This  variation  must  be  low  enough  to  prevent  interference  with  determining  the  proper 
recommendation. 

External  variation  is  that  present  in  the  environment  in  which  the  decision  must  be 
made.  For  example,  a  decision  often  must  be  made  where  each  alternative  may  produce 
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one  of  several  possible  outcomes.  The  probability  distribution  of  the  outcomes  is  a 
source  of  variation  external  to  the  model,  regardless  of  how  the  model  is  constructed. 
Internal  variation  is  a  function  of  the  model  itself.  In  Expected  Utility  Theory,  a  single 
dimensional  utility  function  is  used  to  map  levels  of  some  measure  into  the  unit  interval 
preserving  a  decision  maker’s  attitudes  regarding  the  levels  and  associated  uncertainty. 

An  example  of  this  would  be  the  function  u(x)  =  (l/l  0)  -  x  where  x  is  the  number  of  days 
to  complete  a  project,  assuming  that  the  function  captured  the  attitudes  and  preferences  of 
the  decision  maker.  However  preference  functions  must  be  elicited  from  subjects  and  are 
subject  to  elicitation  error.  This  error  is  an  example  of  internal  variation. 

A  taxonomy  of  decision  analysis  model  variance  is  presented  in  Table  1.  External 
variation  is  often  categorized  as:  none,  known,  or  unknown.  Decision  making  under  these 
conditions  is  referred  to  as  being  under  certainty,  risk,  or  uncertainty,  respectively. 

(Some  authors  group  risk  and  uncertainty  in  a  single  category  employing  the  latter  term.) 
These  external  sources  of  variation  must  typically  be  estimated  in  some  fashion.  For 
example,  a  business  decision  may  require  that  a  projection  be  made  for  the  interest  rate 
for  a  loan  in  a  future  time  period.  Likely  the  estimate  will  not  exactly  coincide  with  the 
rate  that  occurs.  These  estimates  are  listed  separately  in  the  taxonomy  as  they  are  clearly 
not  the  external  variation  itself,  but  also  are  not  due  to  the  model  proper,  as  any 
appropriate  model  may  be  applied.  Further,  these  estimates  may  be  generated  by  the 
decision  maker  himself  or  by  separate  subject  matter  experts.  Therefore  these  subjective 
error  sources  may  not  be  directly  discernable. 

The  internal  error,  that  due  to  the  model  itself,  is  type  specific.  A  decision 
problem  may  be  approached  with  a  wide  variety  of  techniques  available  to  the  operations 
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Table  1.  Taxonomy  of  Variation. 


•  External  Sources 

o  No  Variation  (Certainty), 
o  Known  (Risk).  Probabilities  are  known. 

o  Unknown  (Uncertainty).  Probabilities  are  unknown  -  sometimes  referred 
to  as  states  of  nature. 

•  Estimates  of  External  Sources 

o  Random  Error.  Subjective  estimates  of  probabilities  contain  error, 

p  =  p  +  £ 

o  Bias  Error.  Subjective  estimates  where  probabilities  are  consistently  high 
or  low  for  certain  portions  of  the  domain. 

•  Internal  (Model)  Error 

o  Model  Type  Error 

■  MCDM,  LP,  ... 

■  DA 

•  Model  Basis 

o  Formalistic  (Nonnative) 
o  Behavioral  (Descriptive) 
o  Axiomatic  (Rational) 

•  Decision  Maker 

o  Objectives /V alue  Hierarchy  Elicitation  Error 
o  Preference  Function  Elicitation  Error 

■  Preferences 

■  Risk  Attitudes 

o  Relative  Weights  Elicitation  Error 

o  Alternatives 

•  Time 


research  practitioner.  These  techniques  include  linear  and  non-linear  programming 
(optimization),  simulation,  multi-criteria  decision-making  (MCDM),  and  other  techniques 
as  well  as  decision  analysis.  Each  technique  will  engender  its  own  error  sources.  For 
example,  an  infeasible  optimization  problem  may  still  be  solved,  in  a  fashion,  through 
relaxation  of  constraints.  However  MCDM  may  provide  a  better  solution.  When  two 
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methodologies  produce  the  same  rank  ordering  of  alternatives,  they  are  considered 
strategically  equivalent. 

For  this  research,  only  decision  analysis  expected  utility  modeling,  or  multi 
objective  decision  analysis  (MOD A),  was  considered.  Two  major  categories  of  variation 
are  observed.  The  first  is  the  model  basis.  There  are  three  model  categories.  Formalistic 
models  are  strictly  mathematical.  Referred  to  as  nonnative  models,  they  are  the  standard 
against  which  human  preferences  are  measured.  Problems  with  formalistic  models  were 
recognized  in  early  probability  studies,  and,  as  discussed  in  Chapter  II,  led  to  the  St. 
Petersburg  Paradox.  They  will  not  be  considered  further  here.  Descriptive  models  are 
designed  to  describe  how  decisions  are  made,  and  to  predict  what  those  decisions  would 
be.  Such  models  are  useful  in  adversarial  situations  where  knowledge  of  a  competitor’s 
decisions  would  be  useful.  Obviously  they  have  use  in  the  economic  private  sector, 
sports,  and  international  dealings  in  war  and  peace.  However  descriptive  models  do  not 
provide  a  recommendation  of  what  would  in  fact  be  the  best  decision.  Rational  models 
are  designed  to  provide  such  recommendations.  They  are  referred  to  as  rational  because 
they  provide  the  best  decision  given  a  model  of  a  decision  maker’s  attitudes  about  the 
decision  components.  (Again,  inconsistency  is  present  in  the  literature  with  respect  to 
nomenclature.  Some  use  the  tenn  normative  to  represent  rational,  not  formalistic  models 
as  described  above.) 

The  other  internal  error  source  is  that  incurred  from  the  process  of  describing  the 
pertinent  aspects  of  the  decision  maker  himself.  These  aspects  are  not  directly 
observable,  hence  not  directly  measurable.  They  must  be  established  through  elicitation, 
and  the  resultant  estimates  will  have  error  associated  with  them.  The  objectives  of  the 
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decision  maker  must  be  established,  to  provide  the  structure  for  the  balance  of  the 
elicitation.  Single  dimensional  preference  functions  are  elicited  by  asking  questions  that 
are  designed  to  detennine  the  decision  maker’s  preference  ordering  and  risk  attitudes. 

The  relative  importance  of  the  objectives  must  be  assessed.  Ah  these  estimates  are 
sources  of  error. 

An  additional  possible  source  of  internal  error  is  the  process  of  generation  of 
alternatives.  Failing  to  recognize  an  alternative  may  deprive  the  decision  maker  of  best 
solution.  However  this  error  source  is  not  observable  when  comparing  models.  Further, 
Keeney  argues  that  Value-Focused  Thinking  provides  the  best  methodology  for  avoiding 
this  problem  (Keeney,  1992).  Finally,  variations  in  decision  problems  may  be  introduced 
when  considering  the  time  dimension.  It  is  possible  that  the  decision  context  and  also  the 
decision  maker’s  preferences  and  attitudes  may  change  over  time.  The  sources  of 
variation  are  shown  graphically  in  Figure  16. 

Decision  analysis  studies  often  examine  external  sources  of  variation.  Research 
regarding  the  internal  sources  is  often  limited  to  examining  the  preference  weights,  or  to 
contrast  one  model  basis  with  another.  For  example,  critics  of  Expected  Utility  Theory 
employ  descriptive  models  that  better  explain  choices  contraindicated  by  the  decision 
maker’s  axiomatic  preferences.  When  the  preference  function  is  (rarely)  considered  as  a 
source  of  variation,  it  has  not  been  examined  in  the  sense  of  its  contributions  to  overall 
robustness. 
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Detailed  Research  Objectives 


Objective  1.  Examine  Relationship  Between  Single  Dimensional  Value  And 
Utility  Functions.  The  literature  review  in  Chapter  II  summarizes  the  various  positions 
regarding  the  relationship  of  value  and  utility.  Keller  (1985b)  examined  the  relationship 
between  value  and  utility  functions  for  single  dimensional  problems,  and  recommended 
that  the  work  be  extended  into  multiattribute  problems  and  include  repetitive  elicitations 
in  order  to  assess  elicitation  error.  Typical  decision  analysis  empirical  data  involves 
subjects  who  are  not  professionals  in  the  field  of  interest  of  the  problem  and  monetary 
costs.  Military  operational  decision  making 

Objective  1.1.  Determine  Functional  Relationship  of  Single  Dimensional 
Value  and  Utility  Functions.  Decision  analysis  modeling  under  uncertainty  or  risk  has 
been  performed  both  with  single  dimensional  value  functions  or  single  dimensional 
utility  functions.  Various  positions  are  argued  regarding  these  functions.  Some  argue  that 
there  is  no  theoretical  difference  between  these  functions.  Others  assert  that  while 
differing  in  theory,  the  differences  are  not  significant  in  application.  Finally,  some  argue 
that  they  differ  significantly  in  theory  and  in  employment.  Various  functional 
relationships  have  been  proposed.  Based  on  the  work  of  Pratt  (1964a:  353  -  375), 
researchers  have  examined  the  suitability  of  a  transfonn  from  single  dimensional  value 
functions  to  a  utility  function.  These  functions  are  the  linear,  exponential,  logarithmic, 
and  power  functions.  The  only  linear  function  is  when  value  and  utility  are  equal.  The 
remaining  functions  form  curves  that  are  strictly  concave  or  convex,  (depending  on  the 
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coefficients).  This  suggests  the  subject  is  strictly  risk  averse  or  risk  seeking,  respectively, 
over  the  entire  domain.  Empirical  evidence  suggests  that  this  is  not  always  correct  for 
example,  see  McCord  and  de  Neufville,  1983a).  The  sigmoid  function  pennits  both 
concavity  and  convexity  to  occur  within  the  range  but  has  not  been  examined  for 
adequacy  of  representing  empirical  data  for  DA  preference  functions.  Establishing  that 
the  preference  functions  differ  significantly  for  military  decision  makers  is  an  essential 
task  regarding  the  other  objectives. 

Objective  1.2.  Examine  Characteristics  And  Performance  Of  Various 
Common  Decision  Analysis  Elicitation  Methodologies.  McCord  and  de  Neufville 
(1983a)  provide  data  that  suggest  that  the  differences  encountered  in  eliciting 
theoretically  identical  single  dimensional  utility  functions  by  differing  methods  exceeds 
that  of  the  differences  found  between  corresponding  single  dimensional  value  and  utility 
functions.  Kirkwood  (1997b)  also  offers  a  multiattribute  utility  approach  that  converts 
from  the  value  space  to  utility  space  on  a  multidimensional  basis. 

Objective  1.3.  Improve  The  Metric  For  Comparison  Between  Preference 
Functions.  McCord  and  de  Neufville  (1983a:  184)  point  out  that  the  literature  provides 
few  tools  for  comparing  preference  functions.  They  offer  a  method,  “average  absolute 
difference,”  which  compares  two  functions  against  a  reference  function.  When 
comparing  the  degree  to  which  a  preference  function  approximates  another,  another 
measure  used  has  been  root  mean  square  error  (RMSE).  This  metric  weights  each 
observation  equally  even  though  points  near  the  limits  of  the  domain  are  more 
constrained.  The  domain  endpoints  have  commonly  defined  ordinate  values  for  all 


68 


preference  functions.  It  is  logical  that  equal  differences  are  less  likely  closer  to  the 
endpoints,  or  conversely,  that  equal  differences  are  more  significant  near  the  endpoints. 

Objective  1.4.  Examine  The  Subject’s  Risk  Attitudes.  Bell  and  Raiffa  (1988d: 
395)  suggest  that  constant  relative  risk  aversion  is  “an  appealing  idea.”  Relative  risk 
aversion  means  that  the  utility  function  operating  in  the  value  domain,  u  ,  exhibits 
concavity. 

Object  Objective  1.5.  Improve  The  Metric  For  Assessing  Risk  Aversion.  A 

measure  of  local  risk  aversion  was  developed  by  Pratt  (1964a).  However  no  measure  of 
risk  aversion  for  an  entire  evaluation  measure  for  a  subject  has  been  widely  accepted. 

The  basic  measure  is  the  utility  curve  shape  with  concavity  (convexity)(linearity) 
indicating  risk  aversion  (seeking)  (neutrality).  However  this  fails  to  quantify  the  degree 
of  the  risk  attitude  when  the  curve  is  not  strictly  convex  (concave)  (linear).  The  concept 
of  a  preference  area  was  proposed,  defined  as  the  area  under  the  utility  curve  (Kimbrough 
and  Weber,  1994:  627).  However  the  set  of  all  utility  curves  does  not  have  a  bijective 
relationship  with  the  set  of  all  preference  area  values  so  the  measure  cannot  distinguish 
between  different  functions. 

Objective  2.  Establish  Methods  Of  Increased  Efficiency  Of  Decision  Analysis 
Elicitation  By  Considering  Robustness  Of  Employing  Single  Dimensional  Value 
Functions  As  Surrogates  For  Utility  Functions.  Sensitivity  analysis  in  DA  examines 
the  impact  of  uncertainty  and  perturbation  of  model  parameters.  However  review  of  the 
literature  has  not  produced  an  example  of  perturbing  the  preference  function.  This  is  a 
key  concept  when  consider  the  difference  between  value  and  utility  functions.  By 
establishing  when  perturbations  in  the  preference  function  are  significant,  it  is  detennined 
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when  changing  form  one  preference  function  form  (value  or  utility)  to  the  other  is 
significant. 

For  large,  complex  decision  situations,  value  functions  are  often  constructed  with 
the  aid  of  subject  matter  experts.  When  a  utility  model  is  desired,  only  the  value 
functions  that  are  significant  when  perturbed  need  be  converted  to  utility  functions.  Such 
a  process  has  the  potential  to  greatly  reduce  the  time  burden  on  the  decision  maker. 
Because  for  large,  complex  problems  the  decision  maker  typically  has  a  critical  schedule, 
this  fosters  acceptance  of  the  process. 

Objective  2.1.  Establish  When  Value-Utility  Differences  Are  Significant.  As 

discussed  previously,  work  regarding  value  and  utility  functions  focused  on  how  the 
functions  are  related,  or  examined  their  impact  when  the  entire  DA  model  was 
constructed  as  a  value  model  or  as  a  utility  model.  The  idea  of  development  of  a  hybrid 
model,  with  both  single  dimensional  value  and  utility  functions,  is  not  recorded.  Such  a 
hybrid,  when  properly  constructed,  should  function  as  a  surrogate  for  the  utility  model. 

Response  surface  methodology  techniques  have  demonstrated  efficacy  in 
determining  significant  contributors  from  a  set  of  factors  for  a  variety  of  problem  types. 
Bauer,  Parnell,  and  Meyers  (1999)  demonstrated  that  RSM  is  applicable  to  DA  problems, 
enhancing  the  infonnation  provided  during  the  sensitivity  analysis  phase.  RSM  has  not 
been  applied  to  the  set  of  preference  functions. 

Objective  2.2.  Develop  an  Algorithm  for  Employment  of  a  Hybrid  Value- 
Utility  Model.  No  previous  work  has  been  done  in  this  area. 
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Methodology  and  Scope 


Objective  1.  Examine  Relationship  Between  Single  Dimensional  Value  And 
Utility  Functions.  Twenty  subjects,  all  US  Army  soldiers,  were  interviewed.  Provided 
with  a  tactical  situation  requiring  a  decision  and  four  evaluation  measures,  single 
dimensional  value  and  utility  and  multiattribute  utility  functions  were  elicited.  Relative 
weights  for  the  evaluation  measures,  and  some  demographic  data,  completed  the 
elicitation  session.  These  data  permit  construction  of  value,  utility,  and  multiattribute 
utility  DA  models.  Additionally,  unpublished  data  from  a  study  by  Doyle  (1998)  was 
obtained  for  examination  of  value  and  utility  function  relationships. 

Objective  1.1.  Determine  Functional  Relationship  of  Single 
Dimensional  Value  and  Utility  Functions.  Single  dimensional  value  and  utility  and 
functions  were  examined  to  see  which  functional  relationships  performed  best  as 
transforms  between  the  two  preference  functions.  Possible  transformation  functions 
examined  were  linear,  exponential,  power,  logarithmic,  and  sigmoid.  The  best  fitting 
transformation  was  identified. 

Objective  1.2.  Examine  Characteristics  And  Performance  Of  Various 
Common  Decision  Analysis  Elicitation  Methodologies.  Single  dimensional  utility 
functions  elicited  using  the  certainty  equivalent  and  probability  equivalent  methods  were 
compared  to  detennine  elicitation  error.  This  error  was  compared  to  the  differences 
between  the  mean  utility  function  and  the  corresponding  value  function.  The  two  single 
dimensional  utility  functions,  the  value  function,  and  multiattribute  function-based 
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decision  analysis  models  were  investigated  for  each  subject.  Decision  analysis  models 
differences  for  the  differing  preference  function  bases  were  examined. 

Objective  1.3.  Improve  The  Ability  to  Compare  Between  Preference 
Functions.  A  weighted  root  mean  square  error  was  established  to  improve  the  measure 
of  goodness  of  fit  for  Objective  1.2. 

Objective  1.4.  Examine  The  Subject’s  Risk  Attitudes.  The  risk 
attitudes  were  examined  to  see  if  Bell  and  Raiffa’s  conjecture  regarding  constant  relative 
risk  aversion  held.  The  risk  attitudes  were  also  examined  within  each  subject  to  see  if  the 
risk  attitudes  were  consistent  across  evaluation  measures. 

Objective  1.5.  Improve  The  Metric  For  Assessing  Risk  Aversion.  A 
measure  similar  in  concept  to  mean  and  variance  was  constructed  to  improve  the  ability 
to  compare  subjects’  risk  attitudes. 

Objective  2.  Establish  Methods  Of  Increased  Efficiency  Of  Decision  Analysis 
Elicitation  By  Considering  Robustness  Of  Employing  Single  Dimensional  Value 
Functions  As  Surrogates  For  Utility  Functions.  After  establishing  an  algorithm,  it  was 
tested  in  an  automatic  target  recognition  problem.  A  US  Air  Force  engineer  served  as  the 
decision  maker.  Full  value  and  utility  models  were  elicited,  allowing  employment  of  the 
approach. 

Objective  2.1.  Establish  When  Value-Utility  Differences  Are 
Significant.  Standard  response  surface  methodology  and  design  of  experiments 
techniques  were  employed  with  respect  to  the  robustness  of  the  value  function.  An 
exponential  form  for  u  was  assumed  and  the  exponential  coefficient  varied  throughout  a 
reasonable  range.  Analysis  of  results  from  a  screening  experimental  design  provided  a 
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measure  of  significance,  p  value,  and  coefficients  for  a  least  squares  model.  Non¬ 
significant  utility  functions  indicated  that  the  respective  value  functions  were  adequate 
surrogates. 

Objective  2.2.  Develop  an  Algorithm  for  Employment  of  a  Hybrid 
Value-Utility  Model.  The  significant  utility  functions  were  prioritized  based  on  the 
magnitude  of  the  least  squares  coefficients.  The  algorithm  consisted  on  iteratively 
substituting  a  significant  utility  function  for  its  corresponding  value  function.  The 
iterations  are  terminated  when  the  hybrid  model  estimate  for  the  utility  is  sufficiently 
close  to  the  true  utility  or  when  the  rank  ordering  of  the  alternatives  will  no  longer  be 
affected.  The  hybrid  model  then  is  an  adequate  model  of  the  true  utility  model. 


Detailed  Methodology  and  Results 

The  methodology  and  results  for  Objectives  1  and  2  are  presented  in  Chapters  IV 
and  V,  respectively.  Detailed  results  are  contained  in  the  appendices,  along  with  the  raw 
data.  A  summary  of  the  research  is  provided  in  Chapter  VI. 
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IV.  Examination  of  Relationship  of  Value  and  Utility  Preference  Functions 


General 

As  discussed  in  Chapter  II,  opinions  vary  widely  with  regard  to  the  relationship 
between  value  and  utility  functions.  Often,  research  has  employed  money  as  a  metric 
when  assessing  these  functions.  Research  has  also  suggested  that  industries  or 
professional  groups  may  share  common  preference  function  characteristics.  Little 
decision  analysis  research  is  available  about  preference  function  relationships  for  military 
personnel.  Military  decision  making  presents  a  unique  environment  as  typically 
operational  military  decision  making  does  not  consider  monetary  cost.  Most  decision 
analysis  research  involves  a  dollar  cost  aspect,  whether  the  decisions  are  public  or  private 
sector.  Additionally,  military  operational  decisions  rarely  quantify  risk,  an  aspect  of  the 
utility  function  but  not  the  value  function.  There  are  also  military  decisions  that  consider 
monetary  cost,  such  as  acquisition  and  managerial  decisions.  Military  operational 
decision  making  offers  an  area  of  decision  analysis  study  that  is  little  examined,  yet  has 
unique  aspects.  This  chapter  will  describe  the  elicitation  of  preference  attitudes  of 
military  professionals,  improved  analytical  techniques,  and  the  analysis  of  these 
functions. 
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Army  Tactical  Scenario  Value-Utility  Data 


Tactical  Preference  Function  Data.  Data  was  obtained  from  20  Army 
professionals  through  elicitation  sessions.  The  elicited  data  permitted  construction  of 
alternate  models  of  the  subject’s  preferences  in  a  realistic  military  scenario. 

Description  of  the  Elicitation  Sessions.  A  decision  problem  was 
developed  to  be  pertinent  to  military  professionals,  regardless  of  grade  or  area  of 
expertise.  The  problem  was  developed  from  realistic  situations  a  unit  leader  might  face 
in  an  operational  environment.  The  decision  problem  is  an  example  of  administrative 
forward  movement  of  a  recently  deployed  unit  in  the  communications  zone.  The 
communications  zone  is,  by  US  Anny  doctrine,  the  area  to  the  rear  of  the  tactical  units 
conducting  the  fighting.  It  is  not  intended  that  units  in  the  communications  zone 
encounter  enemy  units  or  fires.  Such  encounters  are  possible. 

The  subjects  were  told  that  they  were  in  charge  of  a  tactical  unit,  of  size  and 
organization  commensurate  with  their  (unspecified)  experience  and  background.  Their 
unit  was  located  in  a  marshalling  area  and  was  required  to  move  to  a  forward  tactical 
assembly  area  (TAA).  After  arriving  at  the  TAA  their  unit  would  receive  an  operational 
assignment.  The  move  to  the  TAA,  which  was  nominally  600  miles,  would  be  restricted 
to  a  single  route  during  a  24-hour  movement  window  beginning  twelve  hours  from  the 
present.  The  situation  is  portrayed  in  Figure  17. 

The  subjects  were  told  that  they  would  have  to  select  one  route  from  several 
choices.  Normally  the  decision  maker  communicates  her  concerns  for  a  decision 
situation.  To  facilitate  comparison  between  subjects,  for  this  study,  the  key  values  were 
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Figure  17.  Situational  Graphic. 

instead  presented  to  the  subjects.  They  were  also  told  that,  based  on  the  situation,  the 
pertinent  aspects  of  the  situation  were  force  protection  and  logistical  considerations.  All 
other  potential  considerations  were  either  not  of  significant  impact  (e.g.,  weather)  or  were 
identical  for  all  potential  routes  (e.g.,  enemy  air  threat).  The  force  protection  aspects  of 
interest  were  exposure  to  enemy  artillery  fire  and  exposure  to  enemy  ambushes.  The 
scales  for  these  two  evaluation  measures  are  the  number  of  route  miles  within  range  of 
enemy  artillery  and  the  number  of  choke  points  suitable  for  establishment  of  ambushes. 
The  logistical  impact  characteristics  are  the  amount  of  basic  load  fuel  remaining  after 
completion  of  the  move,  in  percent,  and  the  required  time  to  negotiate  the  route,  in  hours. 
The  subjects  were  told  that  these  measures  were  independent  from  each  other.  For 
example,  while  longer  routes  required  more  fuel  consumption,  some  routes  had  fuel 
resupply  available.  The  considerations  of  merit  were  presented  in  a  value  hierarchy  that 
was  made  available  to  the  subjects  during  the  interview.  (Figure  18.)  The  domains  of  the 
pertinent  evaluation  scales  were  presented  to  the  subjects.  This  information  is 
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Figure  18.  Value  Hierarchy. 


summarized  in  Table  2.  The  data  for  each  route  was  not  presented  to  the  subjects. 
Additional  minor  details  were  presented  to  make  the  scenario  plausible. 


Table  2.  Route  Characteristics. 


Miles  in 
Enemy  FA 
Range 

Potential 
Enemy 
Ambush  Sites 

Percent  of 
Fuel  Basic 
Load 

Remaining 

Total 

Movement 

Time 

Maximum  Value 

500 

50 

100 

24 

Minimum  Value 

0 

0 

0 

12 

The  Subjects.  The  twenty  subjects  were  all  US  Army  soldiers  assigned 
to,  or  in  support  of,  the  2nd  Battalion,  28th  Infantry,  a  Basic  Combat  Training  unit  at  Fort 
Jackson,  South  Carolina.  All  were  volunteers  and  received  no  compensation  for 
participation.  Twelve  of  the  subjects  were  officers;  eight  were  enlisted  soldiers.  The 
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officers  varied  in  rank  from  First  Lieutenant  to  Lieutenant  Colonel.  The  enlisted  soldiers 


varied  in  rank  from  Private  Second  Class  to  Command  Sergeant  Major.  Two  officers  had 
previous  enlisted  service,  one  in  the  US  Navy.  Eighteen  of  the  subjects  were  male;  two 
were  female.  Years  of  service  varied  from  two  to  26  with  a  mean  of  1 1.4.  Three  subjects 
had  spent  part  of  their  career  in  the  Army  Reserve.  All  were  on  active  duty  at  the  time  of 
the  interview.  Thirteen  of  the  subjects  were  from  combat  arms  backgrounds,  two  from 
combat  support,  and  five  from  combat  service  support. 

Elicitation.  The  elicitation  was  accomplished  through  private  interviews 
with  the  subjects.  The  interviewer  employed  a  questionnaire  to  guide  the  session. 

Figures  1  and  2  were  displayed  throughout  the  interview  process  for  the  subject’s 
reference.  Subjects  were  oriented  to  the  interview  with  the  explanation  that  they  would 
be  given  a  tactical  scenario  and  then  asked  sets  of  questions  about  their  preferences  and 
attitudes.  These  sets  of  questions  would  ask  for  similar  information  but  in  different  ways. 
The  subjects  were  told  that  there  were  no  right  or  wrong  answers  in  any  absolute  sense. 
Instead  the  intent  was  to  examine  the  internal  consistency  of  their  answers  despite 
altering  the  question  structure.  They  were  told  for  this  reason  any  assumptions  they  made 
should  be  maintained  throughout  the  interview.  Subjects  were  advised  that  there  was  no 
attempt  to  trick  them.  At  any  time  the  subject  was  free  to  change  any  information 
provided  earlier.  Interviews  took  between  sixty  and  ninety  minutes  to  conduct,  and  took 
place  in  early  2000. 

The  single  dimensional  weights  for  an  additive  value  function  were  elicited 
following  the  procedures  presented  in  Kirkwood  (1997b:  68  -  70).  Then  the  single 
dimensional  value  and  utility  functions  were  elicited.  The  utility  functions  were  elicited 


78 


utilizing  two  methods:  certainty  equivalent  and  probability  equivalent  techniques 
(Clemen,  1996:  474  -  477).  These  functional  elicitations  were  done  in  two  different 
sequences.  Each  subject  either  provided  information  for  his  or  her  value  function, 
followed  by  the  certainty  equivalent  utility  function,  and  finally  the  probability  equivalent 
utility  function,  or  the  order  of  the  first  two  elicitations  was  reversed.  Subjects  were 
assigned  to  a  sequence  at  random.  Then  information  regarding  the  subject’s 
multiattribute  utility  (Kirkwood,  1997b:  155  -  164)  was  elicited.  For  subjects  for  whom 
there  was  no  “swing”  combination  of  attributes  that  provided  ambivalence,  four  different 
certainty  equivalents  were  established  using  values  besides  the  domain  bounds.  Finally 
some  demographic  data  about  the  subject  was  recorded.  The  interviews  lasted  about  an 
hour  per  subject.  Several  subjects  were  briefly  interviewed  a  second  time  to  elicit 
additional  information  to  correct  an  error  made  by  the  interviewer.  The  author  believes 
that  the  correction  did  not  alter  the  integrity  of  the  experiment. 

Tactical  Questionnaire  Results.  The  data  analysis  began  with  fitting  piecewise 
linear  curves  to  the  points  elicited  for  the  value  and  single  dimensional  utility  functions. 
This  permits  visual  examination  of  the  elicitation  results.  An  example  is  provided  in 
Figure  19,  which  shows  the  three  elicited  preference  functions  for  subject  FJ1  for  the 
enemy  artillery  attribute.  In  this  case,  all  three  preference  functions  appear  to  agree  well. 
This  is  in  contrast  to  the  results  shown  in  Figure  20  for  the  same  attribute  but  elicited 
from  a  different  subject.  For  subject  FJ15  the  single  dimension  utility  functions  agree 
well,  but  differ  greatly  from  the  corresponding  value  function.  Figure  2 1  shows  an 
example  of  reasonable  agreement  between  the  value  function  and  the  probability 
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Figure  19.  Single  Dimensional  Preference  Functions  for  Enemy  Artillery,  Subject  FJ1. 


Figure  20.  Single  Dimensional  Preference  Functions  for  Enemy  Artillery,  Subject  FJ15. 


equivalent  utility,  but  poor  agreement  for  the  certainty  equivalent  utility  function  for  the 
ambush  site  attribute. 
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(CE) 
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Figure  2 1 .  Single  Dimensional  Preference  Functions  for  Subject  FJ1 8. 


Additionally,  multiattribute  utility  functions  were  elicited.  When  the  subject 
could  not  accept  any  combination  of  possible  outcomes  employed  in  Kirkwood’s  method 
(1997b:  163),  Kirkwood  states  the  results  may  be  used  to  bound  the  exponential  form  of 
the  multiattribute  utility  function.  Figure  22  shows  the  results  of  this  procedure  for 
subject  FJ18.  Another  interesting  observation  is  that  subject  FJ15’s  responses  indicated 
single  dimensional  utility  functions  that  were  either  essentially  linear  or  were  moderately 
concave.  The  standard  interpretation  of  these  morphologies  is  that  they  indicate  risk 
neutral  and  risk  averse  behavior,  respectively.  However,  the  subject’s  multiattribute 
utility  curve  was  extremely  convex  (Figure  23).  Results  for  each  subject  are  contained  in 
Appendix  B. 

The  elicited  utility  functions  were  compared  to  several  models  of  transfonns 
between  the  value  and  utility  functions.  The  single  dimensional  value  and  utility 
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utility(upper)  — □ — utility(lower) 


Figure  22.  Multi  Dimensional  Preference  Functions  for  Subject  FJ18. 


Figure  23.  Multi  Dimensional  Preference  Functions  for  Subject  FJ15. 
functions,  v(x) ,  uCE  (x) ,  and  uPE  (x) ,  respectively,  were  elicited  at  three  domain  points 
and  linear  interpolation  employed  to  provide  continuous  curves  for  each  evaluation 
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measure  x, .  The  two  single  dimensional  utility  functions,  uCE  and  uPE  ,  were  averaged  to 
produce  a  mean  single  dimensional  utility  function,  utj  (xk )  =  (u^E  {xk )  +  uEE  (xk  ))j2  for 

the  curves  for  each  subject,  i,  and  each  evaluation  measure,  j,  at  point  xk . 

The  examined  models  of  the  relationship  between  the  value  and  utility  functions 
were  those  of  Keller  (1985):  linear, 

ulm(v(x))  =  v(x),  (46) 


exponential, 


logarithmic, 


i  _  „-«(*(*)) 

uex(v(x))=  ■  _C  ,c*  0. 
\  —  e 


«'°g(v(x))  = 


log(v(x)  +  c)-logc 


log 


1  +  c 


c  >  0 


(47) 


(48) 


and  power, 

upow(v(x))  =  v(x)c,c>  0  (49) 

plus  the  sigmoid  function. 

The  sigmoid  function, 

/w=X+e-“  (50) 

maps  from  the  reals  to  the  unit  interval,  so  this  makes  it  attractive.  The  function  may  be 
made  monotonically  increasing  or  decreasing  by  selecting  a  >  0  or  a  <  0 ,  respectively. 
(When  a  =  0,  /(x)  =  0.5.)  The  difficulty  of  employing  the  sigmoid  function  for 


decision  analysis  is  that  the  domain  is 


(-oo?oo)  and  that  it  is  equal  to  /  (0)  =  0.5 .  In 
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order  to  use  this  function,  the  domain  must  be  translated  from  the  attribute  domain  to  a 


domain,  z,  where  /(0)  =  0.5  and 


for  decreasing  levels  preferred, 
for  increasing  levels  preferred 


(51) 


To  facilitate  this,  the  midvalue  xm ,  the  point  that  satisfies  the  relationship  u  (xm )  =  0.5 

should  be  selected  as  the  point  that  will  translate  to  zero.  Unless  the  DM  is  risk  neutral, 
this  will  not  be  the  midpoint  of  the  interval,  but  as  the  function  rapidly  approaches  the 
codomain  limits  this  will  likely  provide  little  error  in  the  tails  and  guarantee  that  the 
function  fits  at  the  midvalue.  By  inspection  the  interval  [-10,  10]  provides  reasonable 
behavior  for  a  wide  range  of  sigmoid  coefficients  a.  By  this,  we  mean  that  the  curves 
rapidly  approach  values  of  zero  or  one  for  a  large  range  of  coefficient  values  a.  However 
when  the  coefficient  is  close  to  zero,  the  function  approaches  the  constant  function 
f{x)  -  0.5  .  Because  of  this,  the  assumption  for  interval  should  be  checked.  If  the 
coefficient  is  in  the  interval  [-0.3,  0.3]  the  fit  will  likely  be  improved  by  extending  the 
domain. 

A  method  for  this  is  proposed: 

1.  Given  the  evaluation  measure  domain  ,  define  the  midvalue  xm  where 

v(xm )  =  0.5  or  u  (xm )  =  0.5  ,  as  appropriate. 

2.  Define  the  sigmoid  function  domain  j^z°,z*  j  .  Initially  select  z°  =  -10  and 
z*  =10. 
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3.  Determine  the  maximum  distance  Dx  from  xm  to  the  endpoints  of 
employing  D,  =  max [xm  -  x°,x  -  xm) . 

4.  Similarly  define  the  distance  D _  on  the  sigmoid  function  domain.  Initially  this  is 
equal  to  10. 

5.  Define  the  ratio  of  the  two  scales  R  =  DjD _  . 

6.  Determine  the  proper  offset  required  to  align  the  zero  values  on  the  two  scales.  If 
xm  -x°  <  x*  -  xm ,  then  s  =  (x*  -  xm )-  [xm  -  x° ),  else  5  =  0. 

7.  Transform  the  evaluation  measure  scale  to  the  sigmoid  function  scale  using 
z,  ={{x,+s)/R)-Dz. 

8.  Fit  the  sigmoid  function  to  the  set  {(z 0 ,  v (x 0 )),  (z 1 ,  v (x 1  )),•••,  (z  ’ ,  y (x 4 ))} . 

9.  Check  that  v(z°),  l-v(z*)<^  or  w(z°),  l-»(z*)<^  where  ^is  the 

acceptable  value  of  tolerance  of  fit  at  the  endpoints.  For  this  research,  a  sigmoid 
endpoint  fit  was  deemed  acceptable  if  S  <  0.1 .  If  the  tolerance  criterion  is  not 
met,  extend  the  interval  of  the  sigmoid  function  domain  by  selecting  new  z°  and 
z*  values  and  return  to  step  three. 

The  sigmoid  function  of  interest  is  then  given  by 

ri(v(z))  =  j/+e.„(l)  (52) 

The  coefficient,  c,  of  each  model  was  selected  to  obtain  the  best  least  squares  fit  of  the 
function,  u (v (jc, ■ )) ,  to  u{x.) ,  for  each  subject.  The  five  fitted  u  (v (xj)  were  examined 

to  see  how  good  was  the  fit  to  u(x). 
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Measuring  Goodness  of  Fit 


Root  Mean  Square  Error.  In  order  to  assess  the  success  of  a  model  in  fitting  the 
elicited  utility  function,  a  measure  of  the  fit  must  be  employed.  Keller  (1985b:  479) 
recommended  employing  Root  Mean  Square  Error  (RMSE)  as  a  measure  of  the 
differences  between  the  two  functions.  In  this  case  the  transformed  utility  function, 

if  (v  (x)) ,  derived  from  the  empirically  based  value  functions  must  be  compared  to  the 
mean  empirical  utility  function,  u{x).  Keller  (1985b)  followed  the  work  of  Dyer,  Farrell, 


and  Bradley  (1973)  and  Fishburn  and  Kochenburger  (1979)  employing  RMSE  to  test  this 
fit.  She  also  proposed  an  “ad  hoc”  criterion  of  accepting  a  fit  when  RMSE  <  0.05  . 

The  RMSE  is  detennined  employing  the  well-known  fonnula 


RMSE  = 


K 

£[m(v(x;.))-m(x,.) 


(53) 


For  the  purpose  of  examining  the  error  of  u  (v(x)) ,  Equation  (53)  becomes 


RMSE" 


vAxJk))~uAxJk) 


where  the  index  type  model  e  {ex,lin,log,pow,sig}  representing  the  exponential,  linear, 
logarithmic,  power,  and  sigmoid  models  described  in  equations  (46)  through  (52), 
/ e  {1,2,..., 20}  is  the  index  for  the  subject, 

je  {1,2, 3, 4}  is  the  index  for  the  evaluation  measure,  and 
k  e  {1,2,. .  ,,K}  is  the  index  for  the  increment. 
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However,  while  RMSE  is  an  attractive  measure  of  fit,  it  possesses  the  displeasing 
characteristic  of  equally  weighting  all  observations.  Utility  functions  by  definition  equal 
zero  at  the  least  preferred  levels  of  the  independent  variable  and  equal  one  at  the  most 

preferred.  Because  of  this,  the  difference  between  u(v(x))  and  u(x)  will  be  more 

constrained  as  x  approaches  the  domain  endpoints.  This  suggests  that  a  weighted  RMSE 
(WRMSE)  that  gives  more  weight  to  the  differences  the  further  from  the  center  of  the 
independent  variable  midpoint  would  be  a  more  accurate  measure.  Such  a  weighted  least 
squares  fit  is  described  by  Maybeck  (1979:  120). 

Weighted  Root  Mean  Square  Error.  Employment  of  a  weighted  root  mean 
square  error  (WRMSE)  approach  required  selection  of  an  appropriate  weighting  function. 
Once  accomplished,  comparison  of  functions  with  WRMSE  proceeds  as  with  RMSE. 

Weighting  Function  Desired  Characteristics.  Designate  a  weighting 
function  for  the  WRMSE  as  j(x) .  Define  the  domain  as  [x0 ,  x  ]  and  the  midpoint 


xm  =  ^-(x*  -  x0 )  +  x0 .  Let  n  represent  the  number  of  observed  data  points.  The  desirable 

characteristics  for  j(x  )  are:  (1)  j  :[x0,x„]  — >  □  *  (Vinogradov,  1993:464);  (2)  that  j(x) 
be  symmetric  about  xm  ;  (3)  that  /  (x)  be  strictly  monotonically  decreasing  on 
( Xq , Xm  ]  (and  this,  with  property  (2),  provides  that  /  (x)  will  be  strictly  monotonically 
increasing  on  \xm,x») ,  further  these  properties  provide  that  /  (x)  will  be  minimized  at 
xm  );  (4)  that  j  (x)  — >  °°  as  x  — >  jc0,  x, ;  and  (5)  j(xm )  =  1 .  Note  that  when  n  =  1 , 


WRMSE  =  RMSE  as  the  error  is  provided  by  the  single  datum  point. 
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Chebyshev  Polynomial  Weighting  Function.  Orthogonal  polynomials 
provide  a  method  of  fitting  curves  without  employing  multiple  regression  techniques 
given  the  condition  of  evenly  spaced  data  (Considine,  1989:  2105).  Orthogonal 
polynomials  Pn  of  degree  n  with  respect  to  the  weight  h  satisfy  the  condition 

f  Pn  (x)Pm  (x)h (x)  =  0;  n  *  m  (55) 

J  Cl 

where  the  weighting  function  h(x)>  0,  x<=  \a,b\  (Hazewinkel,  1991:  30  -  33).  One  of  the 
classic  orthogonal  polynomials  is  the  Chebyshev  (or  Tchebichef,  and  also  sometimes 

Romanized  as  Tschebysheff)  polynomial  of  the  first  type  where  h{x)  =  (l  —  x2 )  and  is 
defined  on  (-1,1)  (Beyer,  1987:  374  -  378).  This  function  is  shown  in  Figure  24. 


Figure  24.  Chebyshev  Polynomial  Type  1  Weighting  Function,  h(x). 


The  function  h(x)  meets  the  characteristics  discussed  above  when  the  domain  is 
transformed  from  xe  [x0,x„]  to  ze  (—  l,l),  employing  the  relationship 


z,.  =  2 


1 .  When  the  domain  is  in  v(x)  vice  in  x  (i.e.,  when  the  function 


under  consideration  is  u  ( v(x))  rather  than  u  (x) ),  the  domain  is  restricted  to  the  unit 
interval.  hi  this  case  the  transform  from  v(x)  to  z  is  simply  z  =  2v(x)-l.  The  points 
z  =  -1, 1 ,  where  h{x )  =  °° ,  are  neglected  as  there  is  no  error  contribution.  This  function 
is  illustrated  in  Figure  25.  Such  an  arrangement  modifies  the  RMSE  to 


X*  X1  x2  ^  z)  z2 

Figure  25.  Relationship  of  selected  values  of  x  and  z. 


Higher  Order  Chebyshev  Polynomial-like  Weighting  Functions. 

Besides  the  Chebyshev  Polynomial  weighting  function  h  (x) ,  functions  of  higher  powers 
in  z  meet  the  criteria  expressed  above.  Other  functions  similar  to  the  Chebyshev  may  be 
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created  by  modifying  the  exponent.  A  family  of  equations,  of  which  the  Chebyshev  is  a 
member,  may  be  expressed  as 


<(z.p)  =  -  '—y  (57) 

(1  -zp)/p 

where  t (z, 2)  =  h(z) .  These  functions  are  illustrated  in  Figure  26  for  several  values  of  p. 

In  order  to  select  from  this  family  of  curves,  it  is  noted  that  an  additional  desirable 
characteristic  for  the  weighting  function  is  that  the  rate  of  change  of  the  weighting 
function  be  the  same  as  that  of  the  utility  function.  This,  in  conjunction  with  the  other 
characteristics  described  above,  provides  the  closest  match  between  the  behavior  of  the 
weighting  function  and  the  preference  function. 


z 

Figure  26.  Members  of  Family  of  Functions  t{z,p)  for  several  levels  of  p. 


Because  of  the  symmetry  of  the  weighting  function,  the  simplest  case  where  the 
rate  of  change  may  be  examined  is  when  K  =  5  .  This  case  is  analyzed  below.  This 
provides  two  data  points  in  each  half  domain  for  slope  comparison,  xl5x2  for  u  (x)  and 

the  corresponding  z,,z2  for  h  (x) .  While  the  slopes  are  clearly  different  between  a  utility 
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function  and  the  weighting  function,  the  ratios  of  the  slopes  may  be  maintained.  It  is 


u'(x2)/  =/?/(z2)/ 

/u'(x ,)  /  ti  (zx)' 


desired  that  v  2y  ,,  x  =  v  2/u'(  \-  Without  loss  of  generality,  assume  increased 


levels  of  x  are  preferred.  Figure  25  shows  the  data  points  and  their  respective  functions. 

Unfortunately  use  of  the  weighting  function  will  be  with  utility  functions  elicited 
from  an  unknown  subject,  so  the  form  of  uy  (v;,  (x))  in  a  future  analysis  is  unknown. 

Kirkwood  indicates  that  the  exponential  functional  form  may  commonly  be  employed, 
and  further  provides  that  the  exponential  constant  for  the  function  is  generally  in  the 

range  [/?0,/?*]  =  [-l/lO*(x„  -x0),l/lO*(x»  —  jc0  )] .  This  provides  a  range  of  reasonable 
values  for  p  .  The  exponential  function  with  these  values,  [/?„,/?«],  as  well  as  for 
A,eutrai  =  00  ,  is  illustrated  in  Figure  27.  Without  loss  of  generality  (as  the  function  is 
symmetric  about  the  /?neutral  curve),  consider  only  0  <  p  <  p„. 

We  wish  to  designate  some  p  ,  call  it  pm  ,  that  is  representative  of  the  family  of 
curves  evinced  by  this  interval  of  exponential  constants.  Selecting  a  pm  is  illustrated  in 
Figure  28.  The  domain  values  corresponding  to  uy  (vy  (x))  =  0.5  for  p>  and  /?neutral  are 

( x  +x  )  / 

xa  and  xm ,  respectively.  Then  xb  =  v  a  m  and  an  exponential  curve  fitted  through 
xb  has  p  =  0.522  =  pm  and  represents  a  measure  of  central  tendency  for  the  interval 


0 <p<  p«. 
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—  P  =  Po 

■a —  p  =  00 
■A- P  =  P * 


Figure  27.  Exponential  Constants  Effects. 


Figure  28.  Illustration  of  Determining  pm. 


For  u  (v(x)) ,  the  function  fitted  through  xb ,  the  domain  is  the  unit  interval  and 
the  exponential  function  is 


u(v(x)) 


-v(x)  / 

1  -e  /o-522 


\—  e  Y-522 


(58) 
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and  the  derivative  of  Equation  (58)  provides  the  slope 


—  «(v(x))  =  2.2465e' 
dx 


'—1.9 1 57  v(jc)— 0.95785) 


(59) 


The  ratio  of  the  slopes  of  the  two  equidistant  points  in  the  right  half-domain  is 
w/(v(x2))  _  u( 0.833)  _  0.388 


i/  {v  (x[ ) )  u'( 0.667)  0.455 


=  0.853  .  The  ratios  of  the  function  t(z,p)  for 


p  =  2, 4, 6,8  are  4.02,  10.2,  35.5,  and  131.5,  respectively.  Plots  of  the  curves  of  these 
functions  are  shown  in  Figure  26.  Increasing  the  parameter p  moves  the  weighting 
function  away  from  the  desired  slope  ratio  as  the  curve  center  is  flattened,  providing  less 
discrimination.  We  conclude  that  the  basic  Chebyshev  weighting  function  h  ( z )  is  the 

most  appropriate  selection  from  the  family  provided  by  Equation  (57). 

With  a  method  for  quantifying  the  differences  between  functions  established,  the 
next  concern  is  the  amount  of  error  inherent  in  the  elicitation  process.  As  with  any 
observed  process,  there  will  be  error  in  the  captured  data.  Estimating  this  error  is 
valuable  before  making  comparisons  of  the  process  between  functions. 

Estimate  of  the  Elicitation  Error.  In  order  to  examine  the  fit  of  various  models 
against  the  elicited  utility  function,  the  inherent  error  of  that  elicited  utility  function  must 
be  considered.  An  opportunity  to  observe  the  elicited  error  is  available  through  the 
analysis  of  the  elicited  single  dimensional  utility  functions.  These  utility  functions  were 
elicited  through  two  different  methods,  the  certainty  equivalent  method  and  the 
probability  equivalent  method.  Each  elicitation  produced  a  set  of  five  points,  three  of 
which  were  unique  to  the  subject.  (The  endpoints  are  defined.)  However  the 
independent  variable  (evaluation  measure)  differed  for  each  set  of  observations.  To 
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observe  the  differences  in  the  function  instantiations,  linear  interpolation  was  employed 
to  provide  a  continuous  curve  for  each  set.  Line  segments  joined  the  known  endpoints  to 
the  elicited  points,  in  sequence.  The  mathematical  mean, 

,  was  then  determined  for  the  curves  for  each  subject,  i, 


and  each  evaluation  measure,  j,  at  point  xk  at  the  Mi  point. 

For  each  evaluation  measure  100  (50  for  ambush)  equally  spaced  observations 
were  made  of  the  two  interpolated  utility  functions.  The  root  mean  square  error  (RMSE) 
and  weighted  root  mean  square  error  (WRMSE)  were  then  determined  using  these 
observations.  For  this  analysis,  the  RMSE  is  provided  by 


RMSE"  = 


(60) 


and  the  WRMSE  by 


WRMSE “ 


k= 1 


Z,h(zj*)luoE(zj*)-u!!E(zj*) 


k= 1 


(61) 


where 


i e  {1,2,..., 20}  is  the  index  for  the  subject, 


ye  {1,2, 3, 4}  is  the  index  for  the  evaluation  measure, 


k  e  {1,2,. . .,  W}  is  the  index  for  the  increment, 

and  CE  and  PE  indicate  elicitation  methods  of  certainty  equivalent  and  probability 
equivalent,  respectively. 
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An  analysis  of  variance  (ANOVA)  was  perfonned  on  the  evaluation  measures  and 
then  the  subjects  for  both  RMSE  and  WRMSE.  The  intent  was  to  detennine  if  any  subset 
of  the  errors  appeared  to  have  differing  means.  The  RMSE  ANOVA  for  the  evaluation 

measures  (all  subjects  grouped),  RMSE ,  is  shown  in  Figure  29.  The  diamonds 

represent  95  percent  confidence  intervals  on  the  RMSES  data.  The  horizontal  bars  near 

the  top  and  bottom  of  the  diamonds  define  the  overlap  interval  (Sail,  Lehman,  and 
Creighton,  2001 :  178  -  179).  Two  diamonds  must  have  the  disjoint  overlap  intervals  if 
the  respective  distributions  differ  at  the  95  percent  confidence  level.  The  lack  of  a 
significant  difference  is  apparent.  The p  value  for  the  ANOVA,  0.403,  confirms  this. 

The  same  holds  for  the  WRMSES  as  shown  in  Figure  30,  where  the  p  value  is  0.560. 

Trimming  the  data  is  discussed  below.  With  the  most  outlying  point  removed,  the 
differences  between  means  are  still  not  significant.  P  values  are  0.928  and  0.978  for 

RMSES  and  WRMSE ,  respectively.  A  significance  level  (alpha)  of  0.05  is  used  in 

this  study. 

This  indicates  that  the  errors  associated  with  the  elicitation  process  were  not 
different  for  the  separate  evaluation  measures,  regardless  of  which  measure  is  employed. 
This  is  both  heartening  and  helpful.  It  suggests  no  biases  with  respect  to  evaluation 
measures.  It  also  supports  the  pooling  of  the  elicited  data  so  that  it  may  be  treated  as  one 
large  sample  when  desired.  The  errors  associated  with  individual  subjects  is  now  of 
interest. 
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Figure  29.  ANOVA  of  Averaged  Utility  Function  RMSE  by  Evaluation  Measure  for  All 

Subjects,  RMSE 'f  . 


Figure  30.  ANOVA  of  Utility  Function  WRMSE  by  Evaluation  Measure  for  All 

Subjects,  WRMSE" j  . 


Analysis  based  on  the  subjects  provided  the  RMSE "  graph  in  Figure  3 1  and  the 
WRMSE "  graph  in  Figure  32.  The  p  values  were  0.0812  and  0.0330,  respectively.  This 

indicates  that  there  are  likely  significant  differences  between  the  means  for  the  subjects. 
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Figure  3 1 .  ANOVA  of  Utility  Function  RMSE  by  Subject  for  All  Evaluation  Measures, 

RMSE" . 


Subject 

Figure  32.  ANOVA  of  Utility  Function  WRMSE  by  Subject  for  All  Evaluation 

Measures,  WRMSE1'. . 


Inspecting  Figure  3 1  and  Figure  32,  we  see  that  there  are  two  points  that  may  be  outliers. 
The  first  is  one  of  the  elicited  evaluation  measures  of  Subject  12.  Subject  12 
demonstrated  very  different  behavior  regarding  the  time  evaluation  measure  for  the  two 
elicitation  procedures.  When  presented  with  the  certainty  equivalent  lottery,  the  subject 
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was  extremely  risk  seeking,  preferring  the  uncertain  alternative  unless  the  certain 
alternative  was  at  the  best  possible  outcome.  The  subject  presented  the  diametric  attitude 
when  presented  with  the  probability  equivalent  lottery.  Then  the  subject  selected  the 
uncertain  alternative  only  when  the  probability  of  the  more  desired  outcome  was  equal  to 
unity.  In  other  words,  for  the  probability  equivalent  lottery,  the  subject  selected  the 
uncertain  outcome  only  when  all  uncertainty  was  removed.  Naturally  under  this 
condition  the  “uncertain”  alternative  provides  a  more  desirable  outcome  than  the  certain 
alternative.  This  is  extreme  risk  aversion. 

Presenting  completely  opposite  risk  attitudes  for  the  two  elicitation  methods  is 
clearly  disturbing.  A  portion  of  the  certainty  equivalent  data  was  elicited  during  a 
second,  separate,  session.  While  the  time  interval  of  several  weeks  between  the  initial 
and  the  follow  up  session  might  have  contributed  to  this  contradiction,  it  is  not  the 
source.  The  subject  was  shown  the  initial  session  data  and  the  data  were  discussed. 

While  the  second  session  data  is  in  consonance  with  the  first  session  data,  within  each 
respective  method,  the  first  session  data  also  demonstrates  the  incompatible  risk  attitudes. 
The  contradiction  was  not  presented  to  the  subject,  as  it  was  not  discovered  until  analysis 
had  begun.  Apparently  either  the  subject  misunderstood,  some  other  factor  affected  his 
risk  attitudes,  or  one  or  more  of  these  methods  fail  with  this  individual. 

A  possible  second  outlier  is  the  ambush  site  evaluation  measure  for  Subject  18. 
During  the  certainty  equivalent  elicitation  the  subject  was  completely  risk  seeking, 
selecting  the  certain  alternative  only  when  it  offered  no  ambush  sites,  the  best  possible 
outcome  on  the  evaluation  measure.  During  the  probability  equivalent  elicitation,  the 
subject  maintained  risk-seeking  behavior,  but  to  a  much  less  pronounced  degree.  This 
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elicitation  session  was  the  longest,  and  the  subject  often  seemed  to  reconsider  the 
problem  and  his  attitudes  seemed  to  evolve  during  the  session.  Other  evaluation 
measures  for  this  subject  showed  disagreement  between  methods  that  is  higher  than  most 
other  subjects  as  well,  suggesting  that  the  subject’s  views  were  less  firmly  established  or 
apparent.  The  plots  of  the  elicited  utility  functions  for  both  these  subjects  are  contained 
in  Appendix  B. 

As  Subject  12  shows  complete  contradiction  for  time,  that  datum  point  will  be 
treated  as  an  outlier.  The  extreme  error  point  of  Subject  18  will  be  retained,  as 
justification  for  elimination  is  not  clear.  The  results  are  shown  in  Figure  33  and  Figure 
34  for  both  error  metrics  with  the  Subject  12  outlier  removed.  The  p  values  then  become 
0.01 1  and  0.012  for  RMSE  and  WRMSE,  respectively.  (Elimination  of  the  Subject  18 
potential  outlier  produces  p  values  of  0.049  for  RMSE  and  0.0012  for  WRMSE.)  The 
differences  between  the  means  for  trimmed  data  set  are  significant.  We  conclude  that 
there  are  significant  differences  between  the  elicitation  errors  across  subjects.  Restated, 
individuals  varied  with  respect  to  the  amount  of  elicitation  error  that  they  exhibited.  The 
subjects  provide  a  sample  of  the  military  decision  making  population.  As  such,  their 
error  statistics  provide  insight  when  considering  criteria  for  comparison  of  preference 
functions. 

As  there  are  no  significant  differences  with  the  error  data  based  on  the  evaluation 
measures,  the  data  will  be  pooled,  less  the  single  outlier,  and  used  for  an  estimate  of  the 
elicitation  error  for  the  subjects  as  a  group.  The  distributions  of  the  RMSE  and  the 
WRMSE  are  shown  in  Figure  35  and  Figure  36,  respectively.  The  moments  associated 
with  these  distributions  are  shown  in  Table  2.  Employing  a  95  percent  confidence  two- 
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Figure  33.  ANOVA  of  Averaged  Utility  Function  RMSE  by  Subject,  RMSE "  ,  Trimmed 

Data. 


Figure  34.  ANOVA  of  Average  Utility  Function  WRMSE  by  Subject,  WRMSE "  , 

Trimmed  Data. 

tailed  interval,  the  RMSE  for  any  elicitation  should  fall  between  0.0941  and  0.123,  and 
the  WRMSE  should  fall  between  0.0796  and  0.1 10.  Interestingly,  for  this  experiment, 
the  RMSE  threshold  of  0.05  employed  by  Keller  does  not  fall  within  the  confidence 
interval.  As  error  below  the  confidence  interval  is  not  normally  a  concern,  the  error 
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Figure  35.  Average  Utility  RMSE'i.  Histogram  and  Box  Plot.  The  Box  Plot  Rectangle 
Indicates  the  Upper  and  Lower  Quartiles,  the  Internal  Bar  the  Median,  and  the  Diamond 

the  Mean. 
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Figure  36.  Average  Utility  WRMSE1‘.  Histogram  and  Box  Plot.  The  Box  Plot  Rectangle 
Indicates  the  Upper  and  Lower  Quartiles,  the  Internal  Bar  the  Median,  and  the  Diamond 

the  Mean. 


Table  3.  RMSE‘i.1  WRMSE".  Distribution  Moments. 


RMSE 

WRMSE 

Mean 

0.10833 

0.09554 

Std  Dev 

0.06340 

0.066773 

Std  Err  Mean 

0.0071329 

0.0075125 

upper  95% 

0.12254 

0.10951 

lower  95% 

0.094137 

0.079598 

One-sided  95% 

0.12004 

0.10688 
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thresholds  may  be  determined  using  a  one-tailed  test.  This  provides  the  criteria  of  0. 120 
for  RMSE  and  0.107  for  WRMSE  for  acceptable  fit  of  utility  functions. 

Returning  to  the  concern  of  fitting  utJ  ( vy  (x))  to  the  corresponding  utJ  (x(> ) ,  the 

number  of  model  fits  for  various  acceptance  criteria  for  RMSE  and  WRMSE  is  shown  in 
Figure  37  and  Figure  38.  The  fonner  shows  how  many  evaluation  measures  have  been  fit 
by  at  least  one  model  as  a  function  of  the  RMSE  or  WRMSE  criteria.  There  are  80  fits 
for  the  20  subjects,  each  subject  with  four  evaluation  measures.  The  latter  figure  shows 
the  total  number  of  model  fits  as  a  function  of  the  RMSE  or  WRMSE  criteria.  There  are 
400  possible  fits,  five  models  tested  per  four  evaluation  measures  for  the  20  subjects. 
Behavior  of  the  two  metrics  is  similar. 


Criterion 


i  WRMSE  ♦  FMSE 


Figure  37.  Model  Fits  of  Evaluation  Measures  for  RMSE/WRMSE  Criteria. 
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Criterion 


i  WFMSE  ♦  RMSE 


Figure  38.  Total  Model  Fits  for  RMSE/WRMSE  Criteria. 


At  the  acceptance  criteria  developed  above,  0.120  for  RMSE  and  0.107  for 
WRMSE,  the  number  of  model  fits  is  summarized  in  Table  4.  Almost  all  categories 
achieved  a  fit.  Roughly  half  of  the  attempts  to  fit  a  model  were  acceptable.  The  number 
of  fits  employing  the  WRMSE  is  lower,  suggesting  that  WRMSE  is  a  more 
discriminating  measure  than  RMSE.  To  test  this  we  assume  that  the  normal 
approximation  to  the  binomial  is  applicable  to  the  total  fit  data.  This  assumption  is  very 
reasonable  because  the  sample  size  is  so  large  ( nRMSE,nWRMSF  =  400  )  for  the  two  error 
measures.  The  null  hypothesis  is  that  the  proportions  n  for  each  error  metric  are  equal, 

H0  •  K RMSE  ~  ft  WRMSE  (62) 

and  the  alternate  hypothesis, 

H A  •  ft  RMSE  ^  ft  WRMSE  (63) 

Under  the  assumption  the  test  statistic  Z  is  a  standard  normal  deviate  and  is  defined  by 
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(64) 


(  P  RUSE  P  WRMSE  ) 


n(i-n) 


- 1 - 

\nRMSE  n  WRMSE  J 


where 


n 


PrMSE^RMSE  P\VRMSEnWRMSE 
nRMSE  nWRMSE 


(65) 


and  Prmse’Pwrmse  are  the  proportion  of  fits  for  the  RMSE  and  WRMSE  metrics, 
respectively,  and  nRMSE,nWRMSE  is  the  sample  size  for  the  RMSE  and  WRMSE  metrics, 


respectively  (Kanji,  1999:  25). 


Table  4.  Model  Fits  at  Recommended  Criteria. 


RMSE 

WRMSE 

Criterion 

0.120 

0.107 

Categories  Fit 

79/80 

98.8  % 

78/80 

97.5  % 

Total  Fits 

219/400 

54.8  % 

211/400 

52.8  % 

Calculating  the  proportions,  pRMSE  =  219/400  =  0.5475  and 
P wrmse  =  21 1/400  =  0.5275 .  The  variable  n  =  0.5375  .  Calculating  Z  using  Equation 
(64)  provides  Z  =  226.91 .  Because  the  critical  value  Z005  =  ±1.64  ,  we  reject  the  null 

hypothesis  and  conclude  that  there  is  a  significantly  greater  acceptance  of  fits  by  the 
RMSE  metric.  This  indicates  that  the  weighted  root  mean  square  provides  a  more 
conservative  measure  of  the  differences  than  the  root  mean  square. 
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Fit  of  Models 


The  fit  of  the  five  models  is  summarized  in  Appendix  B.  The  best  fits  and 
Keller’s  G  values  (see  Equation  (26),  Chapter  II)  are  presented  in  Appendix  B.  A 
summary  of  the  best  fits,  that  met  the  WRMSE  acceptance  criterion,  is  presented  in  Table 
5.  Only  in  a  few  cases  was  an  acceptable  fit  not  achieved.  The  sigmoid  function  was 
very  successful  as  a  model,  fitting  best  in  over  80  percent  of  the  cases.  No  other  models 
achieved  more  than  a  four  percent  rate  of  best  fit.  It  should  also  be  noted  that  the  linear 

model,  w,'"1  ( v(/  (x))  =  v;/(x)  ,  failing  to  fit  indicates  that  utility  and  value  are  not  identical 

for  any  subject  for  each  of  the  evaluation  measures.  Finally,  Table  5  also  shows  the 
number  of  best  fits  for  each  model  if  RMSE  is  used  rather  than  WRMSE.  Inspection 
indicates  that  RMSE  provided  similar  results  except  the  exponential  model  fit  increased 
to  7.5  percent  and  that  the  logarithmic  function  never  fit  best.  Table  6  shows  how  often 
the  various  models  achieved  an  acceptable  (not  necessarily  best)  fit  at  the  selected 
criteria.  Clearly  the  sigmoid  function  was  very  successful.  Exponential  and  power 
functions  had  about  the  same  success  rate.  The  logarithmic  was  less  successful  and  the 
poorest  fit  was  the  linear  function.  This  supports  the  hypothesis  that  the  value  and  utility 
functions  are  different  constructs  and  different  within  a  professional  population  of 
subjects.  Further,  it  indicates  that  approximating  a  utility  function  with  the  corresponding 
value  function  is  a  poor  approximation  in  the  absence  of  other  information. 

The  good  performance  of  the  sigmoid  may  be  in  part  due  to  the  elicitation  over  a 
fixed  domain  for  all  individuals.  This  was  done  to  ensure  comparable  data  between 
subjects.  But  inclusion  of  the  domain  for  which  the  corresponding  codomain  is  outside 
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Table  5.  Number  of  Best  Acceptable  Fits  By  Model  Type. 


RMSE 

WRMSE 

Linear 

0 

0 

Exponential 

6 

3 

Logarithmic 

2 

0 

Power 

3 

3 

Sigmoid 

67 

73 

None 

2 

1 

Table  6.  Total  Acceptable  Fits  by  Model  Type. 


RMSE 

WRMSE 

Number 

Percent 

Number 

Percent 

Linear 

19 

4.75 

17 

4.25 

Exponential 

46 

11.5 

44 

11 

Logarithmic 

31 

7.75 

28 

7 

Power 

50 

12.5 

45 

11.25 

Sigmoid 

78 

19.5 

77 

19.25 

of  the  region  where  v(x)  =  i,ie  (0,1)  may  have  introduced  an  artifact  that  enhances  the 
probability  that  the  sigmoid  would  be  accepted. 


Curve  Shape 

The  utility  curve  shape  is  traditionally  an  indicator  of  the  subject’s  risk  attitude. 
Concavity  is  indicative  of  risk  aversion,  convexity  of  risk  affinity,  and  linearity  of  risk 
neutrality.  Keller  (1985b),  and  others,  also  categorize  some  curve  shapes  into  “s-shaped” 
when  they  exhibit  both  concavity  and  convexity.  A  summary  of  the  number  of 
differently  shaped  curves  resultant  from  the  elicitations  is  presented  in  Table  7.  Most 
subjects  were  either  risk  averse  or  showed  a  mixture  of  risk  aversion  and  risk  affinity. 
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Table  7.  Utility  Function  u(v(x ))  Shapes. 


Concave 

Convex 

Near  Linear 

S-Shape 

Number 

39 

4 

5 

32 

Percent 

48.75 

5.00 

6.25 

40.00 

Figure  39  shows  this  information  graphically  in  a  mosaic  plot  by  subject.  Mosaic 
plots  are  used  to  depict  frequency  information  for  ordinal  and  nominal  data.  The  column 
width  is  proportional  to  the  number  of  observations  for  the  category  and  is  equal  as  used 
in  this  document.  Each  vertical  bar  is  partitioned  into  subregions  representing,  in  this 
case,  curve  shapes,  and  the  length  of  the  subregion  is  proportional  to  the  occurrence  of 
that  curve  shape.  A  single  bar  at  the  right  of  the  plot  provides  the  overall  curve  type 
proportions.  It  can  be  seen  that  instances  of  risk  affinity  and  neutrality  were  spread 
among  the  subjects.  Differences  in  curve  shape  between  subjects  were  nonsignificant 
with  a p  value  of  0.71  when  testing  the  negative  log  likelihood  with  the  Pearson  Chi 
Square  test  (Sail,  Lehman,  and  Creighton,  2001:  247  -  248). 


FJ10FJ12FJ14FJ16FJ18FJ2  FJ3  FJ5  FJ7  FJ9 
Subject 

Figure  39.  Mosaic  Plot  of  Utility  Function  u(v(x))  Shapes  by  Subject. 
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Figure  40  shows  a  mosaic  plot  of  the  same  information  by  evaluation  measure. 
Interpretation  of  the  risk  attitudes  is  problematic  when  considering  the  s-shaped  curves, 
as  the  relative  amounts  of  risk  aversion  and  affinity  are  not  revealed  by  this  summary. 
Simple  risk  aversion  is  greatest  in  the  case  of  ambush.  Artillery  shows  the  greatest  risk 
neutrality  and  simple  risk  affinity.  The  differences  in  proportion  of  curve  shapes  as  a 
function  of  evaluation  measure  is  highly  significant,  with  a  Pearson  Chi  Square  test  p 
value  of  0.0087.  Flowever  the  small  number  of  near  linear  and  convex  cases  makes  these 
categories  sparse  and  below  the  Cochran  criterion  (Sail,  Lehman,  and  Creighton,  2001: 
245),  so  the  results  are  suspect. 


Eval  Meas 


Figure  40.  Mosaic  Plot  of  Utility  Function  u  (v(x))  Shapes  by  Evaluation  Measure. 
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Elicitation  Error  Compared  to  Value-Utility  Differences 


It  has  been  argued  that  elicitation  methodology  imparts  error  greater  than  what  is 
observed  between  value  and  utility  elicitation  approaches,  or 

:(xi)-uPE(xi) |  >  \u(Xj)  —  v(x,-)|  for  most  xf .  Since  uCE  (x° )  =  uPE  (x° )  =  v(x° )  =  0  and 

(x*  )=  uPE  (x*  )=  v(x*  )  =  1 ,  the  argument  presented  above  for  WRMSE  is  germane. 


*C£V 


lCE  ' 


Employ  as  a  WRMS  measure  of  the  difference  between  the  two  elicited  utility 
functions  for  the  ith  subject  and  the  jth  evaluation  measure,  defined 


^■  = 


X  1/ 
\/2 


1  ■!>(**)  hr  )-<%») 


V  *= i 


Similarly,  use  8U  as  a  WRMS  measure  of  the  difference  between  the  mean 


elicited  utility  function  and  the  elicited  value  function,  defined 


(66) 


4 


x  1/ 

V2 


-ith(zj*)[uv(zft)-vi,(zft) 


iih(zjk)k=i 

V  k= 1 


(67) 


The  null  hypothesis  may  be  stated  as  the  elicitation  error  does  not  differ  from  the 
differences  to  be  observed  between  the  value  and  utility  functions,  then  becomes 

(68) 


And  the  alternate  hypothesis 


H0-.£j<8i] 


(69) 
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The  Sy  and  £{j  terms  are  not  independent,  as  the  subject’s  personal  error  affects 

both.  As  the  subjects’  errors  differed  significantly,  the  hypothesis  was  evaluated  with  a 
paired  t  test.  The  results  are  shown  in  Figure  41 .  The  zero  horizontal  line  in  the  plot 
represents  the  point  where  there  are  no  differences  between  St/  and  £tj .  The  solid 

horizontal  line  enclosed  by  the  dashed  lines  indicates  the  mean  difference  between  cT  and 
£tj  enclosed  by  a  95  percent  confidence  interval.  We  conclude  that  there  are  significant 

differences.  The  corresponding p  value  for  the  t  test  is  less  than  0.0001 .  The  WRMS  for 
8n  is  0. 1 9 1  and  for  £l}  is  0 . 1 02 .  We  rej ect  the  null  hypothesis  and  conclude  that  the 

differences  between  the  value  and  utility  functions  are  significantly  larger  than  the 
elicitation  error  for  this  sample. 


Figure  4 1 .  Comparison  of  uu  and  uv  Elicitations.  Paired  t  Test  of  Utility  Function 
Elicitation  Error  (uu)  and  Differences  in  Elicited  Value  and  Utility  Functions  (uv) 
Employing  the  Weighted  Root  Mean  Square  of  the  Differences. 
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Elicitation  Learning  Effects 


The  elicitation  was  perfonned  employing  a  questionnaire  as  a  guide,  with  two 
pennutations  of  the  order  of  elicitations  of  the  preference  functions.  The  two 
pennutations  were  version  A  (value,  certainty  equivalent  utility,  probability  equivalent 
utility  [vuu])  and  version  B  (certainty  equivalent  utility,  value,  probability  equivalent 
utility  [uvu]).  This  provides  the  opportunity  to  check  for  learning  effects.  It  was 
hypothesized  that  if  indeed  utility  is  a  function  of  both  the  decision  maker’s  preferences 
and  her  risk  affinity,  placing  the  value  function  elicitation  first  should  cause  her  to  clarify 
her  thoughts  on  preferences  before  the  latter  two  utility  elicitations. 

This  is  testable  with  the  null  hypothesis 

H0-£uvu=£vuu  (70) 

and  the  alternate 

ha-£uvu*£,uu  (71) 

Where  £  is  as  defined  above  and  the  subscript  indicates  the  elicitation  sequence. 

The  results  of  an  ANOVA  of  the  WRMSE  by  version  are  shown  in  Figure  42. 

The  difference  is  in  the  range  of  significance,  with  a  p  value  of  0.0210.  A  similar 
analysis  on  the  RMSE  data  provided  a p  value  of  0.0536.  Interestingly  the  version  where 
the  value  function  was  elicited  first  had  the  higher  error.  As  will  be  shown  below,  the 
officer  and  enlisted  soldiers  had  differing  amounts  of  WRMSE.  The  elicitation  versions 
were  assigned  randomly;  however  as  a  check  the  version  was  plotted  in  a  mosaic  plot 
against  the  grade  categories  of  officer  and  enlisted  soldier.  This  is  shown  in  Figure  43. 
No  pattern  is  apparent. 


Ill 


0.8 


0.7- 


0.6- 


0.5- 

LU 

03 

1  0.4- 


0.3- 


0.2- 


Questionnaire  Version 

Figure  42.  ANOVA  on  the  Preference  Function  Elicitation  Order.  The  Order  of 
Elicitation  for  Version  A  was  vuu ,  B  was  uvu  . 


Officer/Enlisted 

Figure  43.  Mosaic  Plot  of  Questionnaire  Version  Versus  Pay  Grade  Category. 


The  results  in  Figure  42  are  clearly  affected  by  the  two  possible  outliers.  As 
discussed  above,  elimination  of  one  of  these  points  is  likely  prudent,  the  other  possible 
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but  less  justified.  Eliminating  the  most  severe  outlier  produces  a  p  value  of  0.0214 
forWRMSE  (p  value  of  0.0721  for  RMSE).  This  is  illustrated  in  Figure  44.  Even  when 
discarding  the  potential  outlier  we  reject  the  null  hypothesis  and  conclude  that  there  are 
significant  effects  observed.  (Elimination  of  both  outliers  produces  p  values  of  0.0299 
and  0.139  for  WRMSE  and  RMSE,  respectively.) 


Figure  44.  ANOVA  on  the  Preference  Function  Elicitation  Order  (Trimmed).  The  Order 
of  Elicitation  for  Version  A  was  vuu,  B  was  uvu. 

However  the  one -tail  test  that  a  vuu  elicitation  error  is  less  than  the  uvu 
elicitation,  or  the  null  hypothesis 

H0  •  euvu  >  £vuu  (72) 

and  the  alternate 

'•  £uvu  -  £vuu  (73) 
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provide  a  more  accurate  test  of  learning  effects.  The  p  value  for  the  untrimmed  data  set 
for  WRMSE  is  0.9893  or  0.9895  for  the  data  set  with  the  most  distant  outlier  removed. 
Clearly  we  reject  the  null  hypothesis  that  eliciting  the  value  function  first  provides  for 
learning  and  reduced  elicitation  error.  In  fact  structuring  the  reverse  construct  as  a 
hypothesis  would  be  accepted  with  this  data.  This  effect  remains  unexplained.  However 
for  these  subjects  eliciting  the  value  function  first  led  to  higher  elicitation  error.  We 
conclude  that  learning  effects  were  not  present  as  hypothesized. 


Simultaneous  Equations 

For  the  multiattribute  utility  elicitation,  the  recommended  method  of  Kirkwood 
(1997:  162  -  163)  was  employed.  The  approach  is  to  place  the  evaluation  measures  in 
the  certain  alternative  at  either  their  best  or  worst  levels  ( x0 ;  or  x.t/  for  all  j)  and  vary  the 

combinations  (a  zero-one  programming  approach)  of  the  certain  alternative  evaluation 
measures  at  best  and  worst  levels  until  ambivalence  is  achieved.  The  reference  gamble, 
the  uncertain  alternative,  is  a  50-50  possibility  of  all  evaluation  measures  at  the  most  or  at 
the  least  attractive  levels.  This  is  depicted  in  Figure  45  for  the  tactical  situation.  The 
uncertain  alternative,  labeled  ^4,  has  a  probability  of  0.5  that  the  evaluation  measures  are 
all  at  the  most  preferred  levels  and  a  probability  of  0.5  that  the  evaluation  measures  are 
all  at  the  least  preferred  levels.  The  certain  alternative  B  has  the  evaluation  measures 
undefined,  where  x .  =  x0  j  or  x,  =  xfj  for  all  j. 
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0  Miles  Artillery, 

0  Ambush  Sites, 

1 00  %  Fuel  Remaining, 
(0.5)  1 2  -H  our  Mo ve  Time. 


500  Miles  Artillery, 
50  Ambush  Sites, 

0  %  Fuel  Remaining, 
( 0. 5)  24  -H  our  Mo  ve  Time. 


Miles  Artillery, 
Ambush  Sites, 

%  Fuel  Remaining, 
Hour  Move  Time. 


Figure  45.  Multiattribute  Utility  Elicitation  Graphical  Representation. 


For  the  subjects  where  no  single  combination  of  evaluation  measures  provided 
ambivalence  to  the  reference  gamble,  simultaneous  equations  were  gathered.  Each  of  the 
nine  subjects  who  were  not  ambivalent  to  any  of  the  xoj,xfj  combinations  was  presented 

four  sequential  elicitations,  where  they  were  offered  various  combinations  of  the  levels  of 
the  four  evaluation  measures.  The  levels  were  adjusted  continually  in  the  natural  number 
domain,  e  ^x0y.,x«7]  for  each /,  rather  than  restricting  each  xf  =  {x0/,x*;} ,  until 

ambivalence  with  the  reference  gamble  was  achieved. 

The  elicitation  of  some  combination  of  evaluation  measures  for  alternative  B  such 
that  the  subject  is  indifferent  between  ^4  and  B  provides  a  set  of  evaluation  scores  that  is 
equivalent  in  utility  to  the  uncertain  alternative  A.  By  the  expected  utility  axioms, 
ambivalence  between  alternatives^  and  B  require E^u (A)j  =  E^u (R) J  where  U (X)  is 
the  objective  function  -  the  multiattribute  utility  function,  generally  defined  by 
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(74) 


U{X)  =  YJw,uj(xj) 

7=1 

for  the  J  evaluation  measures,  where  w .  is  the  weight  of  the  /th  evaluation  measure  and 

X  =  {xx,x2,...,Xj}T  .  As  we  have  different  subj ects  and  four  elicitations  per  subj ect, 
further  employ  the  standard  definition  of  the  utility  of  the  uncertain  outcome 

<4(x)  =  2>a(**)  <75) 

7=1 

where  k  indicates  the  elicitation,  one  through  fours,  for  the  /th  subject.  The  utility  of 
alternative  A  is  easily  determined  as  the  utility  of  all  evaluation  measures  at  the  least 
preferred  levels,  X0 ,  is  zero,  and  at  the  most  preferred,  X* ,  equal  to  one.  The  utility  of 
alternative  A  is  given  by 

U{A)  =  PxU  (X0)  +  /?xf/ (X«)  =  0.5(0)  +  0.5(l)  =  0.5  (76) 

where  px  ,/?x  are  the  respective  probabilities  of  realization  of  each  possible  outcome 
under  alternative  A.  As  alternative  A  does  not  change  for  these  four  elicitations, 

U  (X1/t)  =  U  (X2jt)  =  U  (X3jt)  =  U  (X4/t)  =  0.5 .  Because  the  expected  utility  of  both 

alternatives  is  equal,  Eyu  (5)]  =  0.5  . 

The  multiattribute  value  function  V  is 

K(X)  =  £vv,v,(.v,)  (77) 

7=1 

and  providing  for  multiple  elicitations  and  subjects  becomes 

(78) 

7=1 
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Because  E\U  (5)]  =  0.5  for  all  four  elicitations  if  there  is  a  functional  relationship 
between  value  and  utility,  w,  then  U (B)  =  U (V (B)}  =  0.5  for  all  elicitations.  Or 
U(v(B1))  =  U(v(B1))  =  U(v(B3))  =  U(v(B4))  =  0.5  where  Bk  indicates  the  Mi 
elicitation  iteration.  This  requires  that  V{BX)  =  V  (B2 )  =  V  (B3 )  =  V  (B4 )  =  d  where  d  is 
some  constant. 

Each  elicitation  of  a  combination  of  x  ,  provides  an  opportunity  to  estimate  d.  A 

histogram  and  q-q  plot  is  shown  in  Figure  46.  The  data  is  well  behaved  enough  to 
assume  normal  distribution  of  the  error.  A  95  percent  confidence  interval  extends  from 
0.41  to  0.54,  including  the  nominal  value  of  0.5.  ANOVA  results  for  these  subjects  are 
shown  in  Figure  47.  At  least  one  subject  has  a  significantly  different  error  distribution 
with  a  p  value  of  0.037. 


Normal  Quantile  Plot 

Figure  46.  Histogram  and  Q-Q  Plot  of  the  Simultaneous  Equation  Value  Results. 
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1 


Figure  47.  ANOVA  of  Simultaneous  Equation  Value  Results. 


To  compare  the  error  in  the  multiattribute  case,  eM  ,  to  that  of  the  single 
dimensional  elicitations,  WRMSE  is  again  appropriate.  Each  of  the  four  Bk  for  each 
subject  provides  a  realization  of  an  estimate  of  the  true  curve  that  corresponds  to 
U (V ( Bk )) .  The  four  0  (V (Bk ))  curves  were  determined  by  Kirkwood’s  method  where 
an  exponential  constant  is  calculated  that  causes  the  curve 


U(V(B)) 


1  -  e 


I  /  5 

'P 


P*' 


\  —  e‘ 

v(x),  otherwise, 


(79) 


to  pass  through  the  particular  point  (VM ,  0.5) .  As  it  is  known  that  the  alternative  B  has  a 
utility  equal  to  0.5,  the  point  VM  is  the  point  that  provides  U(VM )  =  0.5  and  is 

determined  as  provided  by  Kirkwood  (1997:  162  -  164)  using  a  simple  look-up  table  for 
p  .  This  table  is  provided  in  Table  8.  Each  Bk  provides  a  separate  p  and  therefore  a 
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separate  estimate  of  the  multiattribute  utility  function,  0  (V  (z,)) .  The  error  of  the 
multiattribute  utility  function  for  each  0  (V  (z7))  is  provided  by 


-,1/ 


Z*(zfi 


U(V(z:))-U(z:) 


k=  1 


k=l 


(80) 


Z  Ujz,) 

where  U(z,)  =  — -  .  The  £M  for  the  kt h  elicitation  and  /'th  subject  is  denoted  £M 


The  £Mk  results  are  summarized  in  Table  9  along  with  the  single  dimensional  elicitation 
error,  £jk ,  as  defined  above,  for  comparison.  Results  of  an  ANOVA,  blocked  on  the 

subjects,  are  presented  in  Table  10.  In  only  two  cases  were  the  single  and  multiple 
dimensional  elicitation  error  significantly  different.  For  one  of  these  two  subjects  the 
single  dimensional  WRMSE  was  higher,  and  the  reverse  was  true  for  the  other  subject. 
The  data  are  inconclusive  as  to  whether  one  method  provides  superior  accuracy. 


Decision  Analysis  Results 

The  decision  analysis  models  for  the  preference  functions  were  compared  using  the  DPL 
programming  package.  The  decision  tree  and  influence  diagram  for  the  tactical  problem 
posed  in  the  questionnaire  are  shown  in  Figure  48  and  Figure  49,  respectively.  There  is 
only  a  single  decision,  that  of  which  route  to  select.  The  uncertainties  are  conditional 
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Table  8.  Exponential  Constants,  p  (Kirkwood,  1997b:  69). 


*0.5  R 

^0.5  ft 

0.00 

0.25  0.410 

0.01  0.014 

0.26  0.435 

0.02  0.029 

0.27  0.462 

0.03  0.043 

0.28  0.491 

0.04  0.058 

0.29  0.522 

0.05  0.072 

0.30  0.555 

0.06  0.087 

0.31  0.592 

0.07  0.101 

0.32  0.632 

0.08  0.115 

0.33  0.677 

0.09  0.130 

0.34  0.726 

0.10  0.144 

0.35  0.782 

0.11  0.159 

0.36  0.845 

0.12  0.174 

0.37  0.917 

0.13  0.189 

0.38  1.001 

0.14  0.204 

0.39  1.099 

0.15  0.220 

0.40  1.216 

0.1C  0.23G 

0.41  1.359 

0.17  0.252 

0.42  1.536 

0.18  0.269 

0.43  1.762 

0.19  0,287 

0.44  2.063 

0.20  0.305 

0.45  2.483 

0.21  0.324 

0.46  3.112 

0.22  0.344 

0.47  4.157 

0.23  0.3C5 

0.48  6.243 

0.24  0.387 

0.49  12.497 

20.5  R 

20.5  R 

0.50  Infinity 

0.75  -0.410 

0.51  -12.497 

0.76  -0.387 

0.52  -6.243 

0.77  -0.365 

0.53  -4.157 

0.78  -0.344 

0.54  -3.112 

0.79  -0.324 

0.55  -2.483 

0.80  -0.305 

0.56  -2.063 

0.81  -0.287 

0.57  -1.762 

0.82  -0.269 

0.58  -1.536 

0.83  -0.252 

0.59  -1.359 

0.84  -0.236 

0.60  -1.216 

0.85  -0.220 

0.61  -1.099 

0.86  -0.204 

0.62  -1.001 

0.87  -0.189 

0.63  -0.917 

0.88  -0.174 

0.64  -0.845 

0.89  -0.159 

0.65  -0.782 

0.90  -0.144 

0.66  -0.726 

0.91  -0.130 

0.67  -0.677 

0.92  -0.115 

0.68  -0.632 

0.93  -0.101 

0.69  -0.592 

0.94  -0.087 

0.70  -0.555 

0.95  -0.072 

0.71  -0.522 

0.96  -0.058 

0.72  -0.491 

0.97  -0.043 

0.73  -0.462 

0.98  -0.029 

0.74  -0.435 

0.99  -0.014 

probabilities  affected  by  the  route  selection.  These  probabilities  were  modeled  as 
triangular  distributions,  with  the  parameters  set  as  listed  in  Table  11. 

The  results  of  the  analysis  are  tabulated  in  Appendix  B.  The  results  were 
calculated  for  each  subject  by  route  for  the  various  preference  functions:  value 


function,  v(x) ;  utility  function  detennined  by  the  certainty-equivalent  method,  uCE  (x) ; 


utility  function  detennined  by  the  probability-equivalent  method,  uPE  (x) ;  utility  function 
determined  by  averaging  the  certainty-equivalent  and  probability-equivalent  methods, 
u(x) ;  and  the  multiattribute  utility  method,  um  (x) . 

The  route  alternatives  ordered  in  descending  preference  appear  in  Appendix  B. 
The  mismatches  between  pairs  of  various  preference  functional  models  are  presented  in 
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Table  9.  WRMSE  for  Single  and  Multiple  Dimension  Utility  Elicitations. 


Subject 

Single 

Dimensional 

Error 

% 

Multi¬ 

dimensional 

Error 

£Mtk 

Subject 

Single 

Dimensional 

Error 

£lk 

Multi¬ 

dimensional 

Error 

£Mik 

FJ1 

0.0676 

0.0253 

FJ12 

0.0712 

0.0515 

FJ1 

0.0289 

0.0692 

FJ12 

0.0730 

0.0124 

FJ1 

0.0534 

0.0457 

FJ12 

0.195 

0.112 

FJ1 

0.0717 

0.0898 

FJ12 

0.728 

0.160 

FJ4 

0.0256 

0.0734 

FJ13 

0.0773 

0.133 

FJ4 

0.0964 

0.113 

FJ13 

0.0648 

0.184 

FJ4 

0.0528 

0.0417 

FJ13 

0.136 

0.193 

FJ4 

0.0991 

0.0049 

FJ13 

0.0616 

0.124 

FJ6 

0.116 

0.0703 

FJ16 

0.0822 

0.344 

FJ6 

0.129 

0.0122 

FJ16 

0.0725 

0.174 

FJ6 

0.157 

0.0917 

FJ16 

0.120 

0.174 

FJ6 

0.0333 

0.153 

FJ16 

0.0309 

0.0469 

FJ9 

0.143 

0.0747 

FJ18 

0.148 

0.135 

FJ9 

0.150 

0.180 

FJ18 

0.486 

0.184 

FJ9 

0.0956 

0.0252 

FJ18 

0.200 

0.0316 

FJ9 

0.153 

0.0811 

FJ18 

0.116 

0.0248 

FJ10 

0.107 

0.036 

FJ10 

0.0373 

0.0077 

FJ10 

0.0616 

0.0065 

FJ10 

0.0575 

0.0348 

Table  10.  Comparison  of  WRMSE  for  Multi-  and  Single  Dimensional  Utility 

Elicitations. 


Subject 

Mean  Single 
Dimensional 
WRMSE 

Mean 

Multidimensional 

WRMSE 

P  Value 

FJ1 

0.0554 

0.0575 

0.907 

FJ4 

0.0685 

0.0583 

0.737 

FJ6 

0.109 

0.0818 

0.522 

FJ9 

0.135 

0.0900 

0.245 

FJ10 

0.0658 

0.0213 

0.0378 

FJ12 

0.267 

0.0840 

0.296 

FJ13 

0.0848 

0.159 

0.0241 

FJ16 

0.0764 

0.185 

0.140 

FJ18 

0.237 

0.0940 

0.175 
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Figure  48.  Tactical  Scenario  Decision  Tree. 
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Figure  49.  Tactical  Scenario  Influence  Diagram  for  the  Value  Function. 
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Table  11.  Triangular  Distribution  Parameters. 


Field  Artillery  Exposure  (miles) 

Route 

Minimum 

Mode 

Maximum 

A 

0 

1 

50 

B 

0 

50 

300 

C 

150 

300 

400 

D 

200 

400 

500 

E 

300 

499 

500 

F 

200 

499 

500 

Ambush  (sites) 

Route 

Minimum 

Mode 

Maximum 

A 

15 

30 

45 

B 

25 

45 

50 

C 

25 

49 

50 

D 

20 

35 

50 

E 

0 

5 

10 

F 

0 

1 

10 

Fuel  (j 

rercent) 

Route 

Minimum 

Mode 

Maximum 

A 

0 

20 

40 

B 

0 

1 

40 

C 

0 

20 

50 

D 

0 

40 

50 

E 

50 

99 

100 

F 

0 

50 

100 

Time  (hours) 

Route 

Minimum 

Mode 

Maximum 

A 

18 

21 

24 

B 

16 

23.9 

24 

C 

17 

22 

24 

D 

14 

21 

23 

E 

12 

12.1 

15 

F 

12 

12.1 

16 

Table  12.  These  mismatches  occur  when  the  different  functional  fonns  produce  differing 
alternatives  in  corresponding  ordinal  positions.  For  example,  if  the  value  function  model 
indicates  that  route  C  is  the  third  most  preferred  alternative  and  the  certainty  equivalent 
utility  model  produces  route  D  for  the  same  position,  this  is  classified  as  a  mismatch.  As 
there  are  six  alternatives  and  six  assignments  per  subject  per  model  type,  the  possible 
number  of  mismatches  per  comparison  is  drawn  from  the  set  {0,2, 3, 4, 5, 6}  .  Also 
contained  in  Table  12  is  the  number  of  mismatches  for  a  deterministic  situation  where 
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Table  12.  Mismatches  Between  Various  Preference  Functions. 


Stochastic 

Deterministic 

Comparison 

Mismatches 

Percent 

Mismatches 

Percent 

uCE{x)-uPE(x) 

28 

0.23 

22 

0.18 

v(x)-uCE(x) 

36 

0.30 

52 

0.43 

v(x)-uPE(x) 

33 

0.28 

49 

0.41 

v(x)-u(x) 

37 

0.31 

56 

0.47 

v{x)-umi  ix) 

12 

0.10 

0 

0.00 

Umx  {x)-unt2  (x) 

6 

0.11 

0 

0.00 

uCE(x)-u(x) 

18 

0.15 

8 

0.067 

ll PE  (x)  -u[x) 

18 

0.15 

14 

0.12 

u{X)-UmXX) 

29 

0.24 

56 

0.47 

“( X)-UmAX ) 

17 

0.32 

25 

0.46 

there  is  no  uncertainty  and  the  route  data  employed  was  the  modal  data  from  the 
triangular  distributions  of  the  stochastic  set  of  cases. 

Normally  in  employment  of  decision  analysis  the  interest  is  in  the  best  (greatest 
value  or  utility)  alternative,  or  in  those  alternatives  near  the  top.  Table  13  shows  the 
mismatches  for  the  best  (highest)  alternative,  and  the  best  two  alternatives.  Mismatches 
for  the  latter  category  are  the  number  of  mismatches  of  the  nonordered  set  [ava2] 

foreach  method  being  compared,  where  the  ai  are  the  alternatives  and  the  subscript 

indicates  order  of  desirability  based  on  the  model.  The  percentage  error  decreases  for  the 
majority  of  cases  when  two  alternatives  are  considered.  This  is  expected  as  the  number 
of  possible  combinations  is  reduced. 

Examining  the  detenninistic  cases  first,  it  is  noticed  that  no  differences  exist 
between  the  value  and  multiattribute  utility  comparisons  and  also  for  comparisons 
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Table  13.  Mismatches  Between  Various  Preference  Functions. 


Best  Alternatives 

Best  Two  Alternatives 

Comparison 

Mismatches 

Percent 

Mismatches 

Percent 

uCE{x)-uPE{x) 

1 

0.05 

4 

0.10 

v(x)-uCE(x) 

5 

0.25 

7 

0.18 

v(x)-uPE(x) 

5 

0.25 

5 

0.13 

v(x)-u(x) 

6 

0.30 

5 

0.13 

vix)-uml  ix) 

1 

0.05 

3 

0.075 

Umx  {x)-unt2  (x) 

0 

0 

2 

0.11 

uCE(x)-u(x) 

2 

0.10 

3 

0.075 

ll PE  (x)  -u[x) 

1 

0.05 

2 

0.05 

5 

0.25 

2 

0.05 

w( X)-UmAX ) 

3 

0.33 

2 

0.11 

between  the  multiattribute  utility  functions  for  the  same  subject  (where  they  exist).  In  the 
deterministic  situation,  the  multiattribute  utility  merely  provides  a  transform  that 
preserved  the  ordinality  of  the  value  data,  so  no  mismatches  are  possible.  Obviously 
employing  multiattribute  utility  in  a  non-aleatory  environment  does  not  provide  any 
additional  ordinal  information  over  that  provided  by  a  value  function.  Clearly  the 
concept  of  employing  value  and  single  dimensional  utility  functions  interchangeably  is 
not  supported  by  these  data.  The  mismatches  between  these  models  almost  reached  half 
of  the  recommendations. 

For  the  stochastic  data,  the  value  and  single  dimensional  mismatches  occur  in 
about  30  percent  of  the  cases.  The  two  realizations  of  the  single  dimension  utility  curve 
produced  mismatches  in  23  percent  of  the  pairs.  The  averaged  single  dimensional  utility 
and  the  multiattribute  utility  differed  almost  one-quarter  of  the  time. 
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Examination  of  the  mismatches  only  provides  a  cursory  insight,  as  the  binomial 
result  contains  no  measure  of  the  distance  of  the  differences  of  the  alternatives.  One 
question  of  interest  is  whether  employing  a  single  elicited  realization  of  the  single 
dimensional  utility  function  significantly  affects  the  results.  Figure  50  shows  the 
pairwise  comparison  of  uCE  (x)  and  uPE  (x)  for  the  alternatives  grouped  by  subjects.  The 

differences  approached  significance  with  a p  value  of  0.0554.  The  interpretation  is  that 
either  preference  realization  may  have  been  employed,  but  only  barely. 


Figure  50.  Pairwise  Comparison  of  uCE(x)  and  uPE(x )  by  Subject. 


Figure  5 1 ,  illustrating  a  pairwise  comparison  between  v  (x)  and  u  (x) ,  grouped  by 


subject,  shows  that  there  is  a  significant  difference  between  them,  with  a p  value  of 
<0.0001.  This  is  no  surprise,  as  the  value  scores  and  utility  scores  were  expected  to 
differ.  The  important  consideration  is  if  the  scores  are  coherent,  producing  similar 
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Figure  51,  Pairwise  Comparison  of  v(x)and  u  (x) . 


recommendations  (identical  ordering,  or  strategically  equivalent  recommendations).  As 
discussed  above,  almost  a  third  of  the  highest  scoring  alternatives  differed  between  these 
preference  measures. 

Figure  52  illustrates  the  pairwise  comparison  between  u(x)  and  u  (x) ,  grouped 

by  subject,  for  the  eleven  subjects  who  did  establish  an  ambivalence  u  (x) .  The  two 

methods  did  produce  significantly  different  scores,  with  a p  value  of  0.0018.  This 
indicates  that  the  multiattribute  utility  construct  is  not  identical  with  the  single 
dimensional  utility  preference  model,  at  least  possessing  differing  means.  This  result 
alone  does  not  indicate  that  the  multiattribute  utility  model  (or  the  single  dimension 
utility  model,  for  that  matter)  is  not  acceptable,  but  that  the  subjects  produced  differing 
constructions  for  these  two  utility  functions.  As  indicated  above,  the  multiattribute  utility 
function  produced  24  percent  mismatches  with  the  averaged  single  dimension  utility 
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Figure  52.  Pairwise  Comparison  of  u(x)  and  w)H|  (x) . 

function  model.  However  the  two  elicited  single  dimension  utility  functions  mismatched 

in  23  percent  of  the  alternatives.  For  five  of  twenty  of  the  subjects,  the  comparison 
between  u  (x)  and  unh  (x)  produced  differing  answers  for  the  best  route.  When 

comparing  uCE  (x)  and  uPE  (x)  for  the  best  alternative,  there  was  only  a  single 

disagreement.  This  indicates  that  the  two  models  are  in  fact  different  constructs, 
measuring  the  subject’s  predilections  differently,  and  which  produce  solution  vectors  that 
are  not  strategically  equivalent. 
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Tactical  Scenario  Value  and  Utility  Preference  Function  Summary 

The  preference  function  information  of  20  subjects,  all  military  professionals,  was 
examined  with  respect  to  differences  in  single  dimensional  value  and  utility  functions.  A 
new  measure  of  comparing  preference  functions,  the  weighted  root  mean  square  error 
(WRMSE),  was  introduced.  The  fit  of  five  proposed  transfonns  from  value  into  utility 
were  examined.  Acceptance  criteria  were  developed  and  tested.  Previous  work  had  used 
ad  hoc  criteria. 

The  sigmoid  function  was  very  successful  as  a  model,  fitting  best  in  over  80 
percent  of  the  cases,  with  other  models  achieving  no  more  than  a  four  percent  rate  of  best 
fit.  The  analysis  also  conclusively  confirms  that  value  and  utility  are  different  functions 
and  are  poor  surrogates,  when  individually  considered,  for  each  other,  at  least  for  these 
data.  The  differences  in  the  value  and  utility  functions  were  significantly  greater  than  the 
elicitation  error.  The  methods  of  probability  equivalent  and  certainty  equivalent  utility 
elicitation  provide  (barely)  similar  results. 

Curve  shape  is  indicative  of  risk  attitudes  when  risk  is  included  in  the  assessment 
protocol.  The  subjects  were  risk  averse  for  half  of  the  evaluation  measures.  They 
provided  s-shaped  utility  curves  in  40  percent  of  the  cases,  indicating  both  risk  averse  and 
risk  seeking  behavior  across  the  evaluation  measure  domain.  The  common  assumption  of 
risk  aversion  is  poor  for  this  problem  and  group  of  subjects,  and  suggests  that  this  may  be 
common  for  military  decision  making  in  operational  settings. 

Order  of  elicitation  of  single  dimensional  value  and  utility  infonnation  made 
significant  differences  in  the  elicitation  error.  Eliciting  functions  in  the  sequence  of 
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value,  utility  (certainty  equivalent),  then  utility  (probability  equivalent)  had  higher 
internal  error  than  utility  (certainty  equivalent),  value,  and  then  utility  (probability 
equivalent).  If  it  is  accepted  that  the  probability  equivalent  and  certainty  equivalent 
methods  produce  theoretically  identical  constructs,  then  this  supports  the  idea  that 
eliciting  value  in  between  the  two  utility  elicitations  provides  reduced  elicitation  error 
than  eliciting  the  utility  infonnation  sequentially  without  interruption.  Such  an  idea  is 
counterintuitive,  and  suggests  future  work  in  this  area.  In  any  case,  typical  application  of 
decision  analysis  involves  the  use  of  only  one  method,  and  so  this  has  no  effect  on 
practical  use  of  DA. 

The  multiattribute  utility  approach  of  Kirkwood  is  significantly  different  than  the 
standard  single  dimension  utility  approach.  Nevertheless,  the  two  methods  provide 
similar  levels  of  accuracy.  When  eliciting  single  dimensional  utility  functions,  either  the 
certainty  equivalent  or  the  probability  equivalent  techniques  may  be  employed,  but  the 
methods  differ  significantly  with  statistical  difference  approaching  significance,  as  the  p 
critical  value  was  0.055. 


Information  Operations  Value-Utility  Data 

Information  Operations  Value-Utility  Functions.  Doyle  (1998)  employed  a 
decision  analysis  value-focused  thinking  approach  to  develop  evaluation  measures  for  the 
analysis  of  information  operations.  This  effort  was  the  first  use  of  decision  analysis  in 
information  operations.  Doyle’s  published  research  was  based  on  single  dimensional 
value  functions.  However  both  value  and  utility  functions  were  elicited  from  participants 
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in  the  study  and  so  the  data  offer  the  opportunity  to  compare  single  dimensional  value 
and  utility  functions  elicited  from  the  same  decision  makers  who  are  professionals 
dealing  with  a  operational  military  subject.  (Doyle,  2000)  These  opportunities  are  rare, 
and  provide  the  opportunity  to  validate  the  above  results  with  data  obtained  from  a 
separate  researcher  and  separate  subjects.  The  background  information  presented  in  the 
next  section  provides  a  contextual  summary  as  described  by  Doyle. 

Background.  Information  Operations  are  defined  by  Joint  Publication  3-13 
(Department  of  Defense,  1998: 1-9)  as  those  “actions  taken  to  affect  adversary 
information  and  information  systems  while  safe  guarding  one’s  own  information  and 
information  systems.”  The  intent  is  to  affect  the  opponent’s  capabilities  and  actions 
through  influences  on  their  infonnation  and  information  systems,  which  provides  a  more 
efficient  defeat  mechanism,  at  least  under  some  circumstances,  than  more  traditional 
methods.  During  periods  of  conflict,  information  operations  are  referred  to  as 
information  warfare  (Department  of  Defense,  1998: 1-1 1). 

Command  and  control  warfare  is  an  application  of  infonnation  warfare  in  military 
operations  and  consists  of  five  components.  Operations  security  is  a  process  of 
identification  of  critical  information,  and  execution  of  corresponding  safeguards  to 
reduce  enemy  exploitation  of  this  information  (Department  of  Defense,  1997:  GL-2). 
Military  deception  consists  of  those  actions  designed  to  deceive  an  enemy  and  causes  the 
enemy  to  undertake  actions,  or  fail  to  do  so,  that  facilitate  friendly  success  (Department 
of  Defense,  1996c:  1-1).  Psychological  operations  convey  selected  infonnation  to  foreign 
audiences  to  create  conditions  favorable  to  friendly  operations  (Department  of  Defense, 
1996a:  1-1).  Electronic  warfare  is  the  use  of  electromagnetic  or  directed  energy  devices 
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to  control  the  electromagnetic  spectrum  or  attack  the  enemy.  Electronic  warfare  has  three 
subcomponents:  electronic  attack  engages  enemy  units  and  facilities  to  degrade  their 
capability;  electronic  protection  safeguards  friendly  electromagnetic  systems;  and 
electronic  warfare  support  gathers  infonnation  from  electromagnetic  emissions  to  provide 
information  about  enemy  forces.  (Department  of  Defense,  2000: 1-2  -  1-3).  Physical 
destruction  is  the  use  of  physical  weapons  against  enemy  targets  as  an  element  of 
command  and  control  warfare  (Department  of  Defense,  1996b:  II-7). 

While  historical  examples  of  what  is  now  known  as  information  operations  are  as 
old  as  military  history,  the  creation  of  the  field  of  information  operations  is  relatively 
recent.  As  information  operations  span  a  number  of  areas  of  specialization,  those 
charged  with  conducting  them  face  the  challenges  of  mastering  an  immature  discipline 
with  disparate  proficiency  requirements.  It  was  desired  that  a  decision  support  tool  be 
developed  to  assist  in  selection  between  information  operations  courses  of  action. 

Results.  Doyle’s  analysis  provided  both  an  objectives  and  cost  value  hierarchy. 
These  are  seen  in  Figure  53  and  Figure  54,  respectively.  As  an  example,  the  value  and 
utility  cost  functions  for  Minimize  Friendly  at  Risk  Personnel  are  shown  in  Figure  55. 

The  domain  for  each  evaluation  measure  differs,  but  in  Doyle’s  work  all  preference 
functions  map  into  [0, 10] .  Plots  of  all  preference  functions  are  contained  in  Appendix  C. 

Not  all  preference  functions  were  continuous. 

We  now  move  beyond  the  analysis  conducted  by  Doyle.  The  continuous 
functions  were  normalized  by  converting  their  codomain  from  [0, 10]  to  [0,  l] .  Then 
analysis  proceeded  as  above  for  the  Measuring  Goodness  of  Fit  for  the  tactical  scenario 
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data.  Fits  of  the  five  proposed  models  were  examined.  As  there  were  no  multiple 
elicitations  of  the  same  preference  functions,  confidence  intervals  could  not  be 
established  for  WRMSE  as  done  previously.  The  earlier  criterion  was  employed. 
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Figure  55.  Minimize  Friendly  at  Risk  Personnel  Functions. 

The  fit  of  the  five  models  is  tabulated  in  Appendix  C  for  the  twelve  continuous 
evaluation  measures.  The  best  fitting  models,  and  other  acceptable  fits  are  tabulated  in 
Table  14  along  with  Keller’s  measure  G.  The  sigmoid  function  provided  the  best  fit  in 
seven  cases  and  the  exponential  in  the  remaining  five.  Of  the  twelve  continuous 
evaluation  measures,  the  curve  shapes  of  the  assessed  utility  functions  of  the  subject  were 
not  strongly  risk  averse:  two  were  linear  or  near  linear,  four  were  S-shaped,  four  were 
convex,  and  two  were  concave.  The  equivalent  results  employing  RMSE  are  presented  in 
Appendix  C  and  Table  15.  The  sigmoid  function  provided  the  best  fit  in  eight  cases  and 
the  exponential  in  the  remaining  four.  Using  either  measure,  the  linear  fit  was  never  the 
best,  indicating  that  value  and  utility  are  significantly  different  in  this  analysis.  One-third 
of  the  i/(v(x))  curves,  the  transform  from  value  to  utility,  were  S-shaped,  indicating 
non-constant  risk  attitudes. 

We  conclude  that  this  data  set  also  supports  the  hypothesis  that  u  (v(x))  ^  v(x) . 

Further,  the  fit  of  the  sigmoid  function  indicates  that  constant  risk  attitudes  are  not 
present  across  the  domain,  as  well  as  that  the  sigmoid  is  an  efficacious  representative 
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Table  14.  Summary  of  Model  Fits  for  t/(v(x))  Employing  WRMSE.  “Lin”  stands  for 
linear,  “Log”  for  logarithmic,  “Pow”  for  power  functions. 


Best 

Acceptable 

Fit 

G 

Other 

Acceptable 

Curve 

Shape 

Increase  Recovery  Time 

Expo 

0.417267 

Lin,  Log,  Pow,  Sig 

Linear 

Degrade  Update 

Sigmoid 

0.890747 

Lin, Exp, Log, Pow 

S-Shaped 

Increase  Error  Content 

Sigmoid 

0.895757 

Exp, Pow 

Convex 

Age  Devaluation 

Sigmoid 

0.863706 

Exp, Pow 

Convex 

Decrease  Accuracy 

Sigmoid 

0.888913 

Exp, Pow 

S-Shaped 

Reduce  Timeliness 

Expo 

0.884586 

Lin, Log, Pow, Sig 

Near  Linear 

Consume  Essential 
Resources 

Expo 

0.98388 

Lin, Log, Pow, Sig 

Concave 

Consume  Essential  Time 

Expo 

0.866733 

Pow,  Sig 

Convex 

Friendly  at  Risk  Cost 

Sigmoid 

0.931184 

None 

S-Shaped 

System  Survivability  Cost 

Expo 

1 

Pow 

Convex 

Collateral  Damage  Cost 

Sigmoid 

0.90562 

None 

S-Shaped 

Sensitive  Information 
Security  Cost 

Sigmoid 

0.991695 

None 

Concave 

Table  15.  Summary  of  Model  Fits  for  j/(v(x))  Employing  RMSE.  “Lin”  stands  for 
linear,  “Log”  for  logarithmic,  “Pow”  for  power  functions. 


Best 

Acceptable 

Fit 

G 

Other 

Acceptable 

Curve 

Shape 

Increase  Recovery  Time 

Expo 

0.50013 

Lin, Log, Pow, Sig 

Linear 

Degrade  Update 

Sigmoid 

0.931271 

Lin, Exp, Log, Pow 

S-Shaped 

Increase  Error  Content 

Sigmoid 

0.93979 

Exp, Pow 

Convex 

Age  Devaluation 

Sigmoid 

0.919279 

Exp, Pow 

Convex 

Decrease  Accuracy 

Sigmoid 

0.93384 

Exp, Pow 

S-Shaped 

Reduce  Timeliness 

Expo 

0.894591 

Lin, Log, Pow, Sig 

Near  Linear 

Consume  Essential 
Resources 

Expo 

0.985921 

Lin, Log, Pow, Sig 

Concave 

Consume  Essential  Time 

Sigmoid 

0.888864 

Exp, Pow 

Convex 

Friendly  at  Risk  Cost 

Sigmoid 

0.954836 

None 

S-Shaped 

System  Survivability  Cost 

Expo 

1 

Pow 

Convex 

Collateral  Damage  Cost 

Sigmoid 

0.93656 

None 

S-Shaped 

Sensitive  Information 
Security  Cost 

Sigmoid 

0.995113 

Pow 

Concave 

parametric  function  for  u  (v(x)) .  These  results  agree  with  the  findings  of  the  analysis  of 
the  Army  personnel  considering  the  tactical  situation  described  above. 
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Comparing  Preference  Functions 


The  research  question  of  interest  is  how  may  differences  in  risk  attitudes  as 
evinced  by  utility  functions  be  compared?  Specifically,  how  does  the  risk  attitude 
presented  by  a  subject  on  one  evaluation  measure  predict  risk  attitudes  on  other 
evaluation  measures?  If  inferences  are  possible,  then  efficiencies  in  elicitation  may  be 
achieved.  Define  consistent  risk  attitude  as  the  condition  on  the  decision  maker  to 
maintain  risk  attitudes  on  all  evaluation  measures  that  are  consistent  with  respect  to  being 
risk  averse,  risk  neutral,  or  risk  seeking.  If  this  condition  is  present,  then  the  decision 
maker  is  risk  averse  (neutral)  (seeking)  on  all  evaluation  measures  if  she  is  risk  averse 
(neutral)  (seeking)  on  any  single  evaluation  measure. 

In  decision  analysis  a  concave  (convex)  (linear)  utility  transform  function, 
u  ( v  (x)) ,  indicates  risk  aversion  (affinity)  (neutrality).  Decision  analysis  literature 

suggests  that  when  a  decision  maker  is  risk  averse  (seeking)  (neutral)  on  a  single 
evaluation  measure  then  she  will  be  risk  averse  (seeking)  (neutral)  on  all  evaluation 
measures.  Traditionally  these  utility  functions  are  assumed  to  be  strictly  concave, 
convex,  or  linear.  However  s-shaped  curves  are  possible.  Indeed,  s-shaped  curves  are 
common  in  the  results  of  the  tactical  questionnaire  data. 

When  examining  the  risk  attitudes  across  evaluation  measures  within  subjects,  the 
curves  may  be  grouped  by  shape.  Such  a  categorical  grouping  is  rendered  ineffectual  as  a 
risk  attitude  measure  because  s-shaped  curves  demonstrate  both  risk  seeking  and  risk 
averse  behavior.  Grouping  all  s-curves  will  place  those  that  are  strongly  risk  averse  in  the 
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same  taxon  as  those  which  are  strongly  risk  seeking.  Clearly  a  quantified  metric  is 
desirable  over  the  crude  categorization  provided  by  curve  shape  alone. 

As  summarized  in  Chapter  II,  preference  functions  may  be  compared  at  discrete 
points  or  across  the  domain.  Under  some  circumstances  a  researcher  may  be  interested  in 
comparing  preference  functions  at  a  specific  point  or  points,  but  generally  the  interest  lies 
in  the  aggregate.  Often  a  decision  maker  will  be  characterized  as,  say,  risk  averse,  which 
is  in  reference  to  risk  attitudes  across  the  domain. 

One  metric  proposed  for  risk  attitudes  is  the  preference  area.  This  is  the  area 
under  the  utility  curve,  u  (v  (x)) .  A  risk  neutral  DM  would  have  a  preference  area  equal 

to  0.5.  A  risk  averse  DM  would  have  a  preference  area  greater  than  0.5,  and  a  risk  averse 
decision  maker  would  have  a  preference  area  less  than  0.5  When  analyzing  data,  risk 
neutrality  is  defined  as  corresponding  to  the  interval  [0.45,  0.55].  (Kimbrough  and 
Weber,  1994:  627) 

A  shortcoming  with  the  preference  area  is  that  for  a  given  preference  area,  many 
curves  may  be  constructed  that  enclose  the  same  area.  For  example,  the  curves  shown  in 
Figure  56  may  be  generated  with  the  same  area  under  the  curve.  Clearly  the  preference 
area  alone  does  not  uniquely  identify  s-shaped  curves,  as  Figure  56B  and  Figure  56C 
illustrate. 

It  is  desirable  to  have  a  measure  of  risk  attitude  that  clearly  distinguishes  between 
risk  aversion  and  risk  seeking  behavior.  Using  a  sign  change  to  contrast  risk  seeking 
versus  risk  averse  behavior  provides  this.  For  the  utility  functions  operating  in  the  value 

domain,  w(v(j)),we  restrict  the  discussion  to  the  class  of  functions 
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Figure  56.  Alternate  Utility  Curves  With  Same  Preference  Area.  Graph  A,  ramp 
function;  Graph  B,  inverse  sigmoid  function;  Graph  C,  sigmoid  function;  and  Graph  D, 

step  function. 


-«■  =  {/:  [0,1]  — >  [0,1]  I  /  is  piecewise  continuous  and  nondecreasing  on 

[o,i],/(o) =0,/(l)  =  l} . 

As  the  preference  area,  with  its  single  measure,  fails  to  adequately  describe  risk 
attitudes,  consider  that  parametric  statistics  frequently  employ  two  parameters  to  describe 
distributions.  For  example,  the  nonnal  distribution  is  completely  described  by  defining 
the  mean  and  standard  deviation.  Instead  of  comparing  discrete  observations  to  the  mean 
(a  function  with  slope  equal  to  zero),  we  are  interested  in  comparing  these  data  to  the 
corresponding  risk  neutral  point. 

Define  the  risk  neutral  function  n(x)  =  x .  The  difference  in  preference  area  to 
the  risk  neutral  situation  is  indicated  by 

i?(/)  =  jo[/(x)_n(x)]^T  (81) 
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and  is  illustrated  in  Figure  57.  The  shaded  area  is  the  region  measured  by  R  (/) . 


Because  J  n  (x)  =  0.5,  then  R  e 
interval 


1/  1/ 

72’  72. 


To  gain  a  more  appealing  metric  on  the 


Figure  57.  Graphs  of / and  n. 


[-1,1] ,  equation  (81)  is  modified  to 

7?(/)  =  2{o[/(x)-n(x)}/x  (82) 

When  7?  (/)  is  positive,  the  subject  is  exhibiting  risk  averse  behavior.  When  it  is 
negative,  risk  seeking  behavior  is  present.  Accepting  the  risk  neutrality  criteria  of 
Kimbrough  and  Weber  (1994:  627),  neutrality  is  present  when  -0. 1  <  R  (/)  <  0. 1 . 

R  (/)  provides  a  measure  of  risk  attitude,  but  like  the  preference  area,  is  not  unique. 

R  (/)  provides  a  measure  of  central  tendency  like  the  mean.  A  measure  of  spread  away 
would  assist  in  characterizing  the  preference  functions.  Variance  would  compare  /  (x) 
to  mean  value,  a  horizontal  construct.  Such  a  construct  has  no  interpretation  in  decision 
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analysis.  Measure  of  spread  with  reference  to  n(x)  is  straightforward  and  is  a  metric  that 
compares  to  a  function,  n(x) ,  that  has  a  specific  meaning  in  decision  analysis.  Define 

i 

F(/)  =  3J[/(x)_n(x)T^x  (83) 

0 

where  the  coefficient  3  is  employed  to  provide  results  on  [0,  l] . 

The  measures  R  (/)  and  V  (/)  provide  different  results  than  the  preference  area 
metric.  The  graphs  in  Figure  56  all  have  a  preference  area  of  0.5,  but  produce  generally 
different  results  for  R  (/ )  and  V  (/) .  Examples  of  representative  functions  for  those  in 

Figure  56  are  displayed  in  Table  16.  These  metrics  do  discriminate  between  these 
functions  in  most  cases  and  so  this  approach  is  an  improvement  to  the  preference  area 
metric.  Where  the  preference  area  fails  to  discriminate,  so  does  /?(/).  However,  V  (/) 

provides  discrimination  when  the  functions  are  not  reflections  about  n(x) .  The  metric 
R(f)  does  provide  the  aggregate  risk  attitudinal  information,  just  as  the  preference  area 
does. 


The  functionals  R(f)  and  V (/)  would  be  defined  in  the  region  shown  in  Figure 
58.  The  lower  bound  of  possible  data  points  within  the  region  is  the  parabola  defined  by 
Hf)  =  Jy  (/') .  Noting  the  definition  of  R  (/)  as  provided  in  Equation  (82),  we 

note  that 


^(/)^|^(/)|=  2^f(x)-n(x)dx  =2JJ  f{x)-n(x)\fl: 


(84) 


By  the  Schwarz  inequality  (Strang,  1988:  147), 
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Table  16.  Example  Functions  With  Preference  Areas  Equal  To  0.5 


/(*) 


n(x) 

n (x)  =  v(x) 

Convex, 

concave 

g(v(x))  =  - 

1  p~v(x)'p 

x_e-*,,„  .P  =  - 0-U;  xs  (0,0.5) 

Y  _  e-(v(*)-0.5 )/p 

- r— - ,  p  =  0.15;  otherwise 

\  _  e~°-5lP  ^ 

Concave, 

convex 

/i(v(x))  =  < 

1  —  e~v^'p 

l_e-o.5ip  ’  P  =  °-15’  xe  (0,0.5) 

- 7TT7 - ,  /?  =  -0. 1 5;  otherwise 

1  —  e~°'5  p  r 

Step 

t(v(x))  =  {°’  V<X!<05 

V  V  ”  (1,  v(x)  >  0.5 

R{f) 

o 


Figure  58.  Region  Defined  by  R\f)  and  V(f). 


Jo  If  (x)  ~  n  (x)Ky  -  2a/L‘  {2dx  V  Jo  V  (x)  “  n  (x)|~ dx 

2  j0'  i  /  (x)~n  ( ^  ^  2Vio  i/ (^) _  ”  (^)T  ^ 

2Jo  V  (*)  ■ n  (*)^  - 2 


It  may  be  that  a  single  subject  may  provide  elicited  responses  that  are  grouped 
with  in  the  region.  If  so,  it  may  be  possible  to  draw  inferences  about  subject’s  responses. 
For  example,  after  sampling  a  number  of  evaluation  measures  for  a  subject,  it  may  be 
inferred  at  some  level  of  confidence  that  the  additional  measures  will  lay  within  some 
radius  about  the  centroid  of  the  current  observations.  If  this  is  correct,  then  this  may 
assist  in  efficiency  of  elicitations. 

The  data  from  the  tactical  questionnaires  was  analyzed  to  detennine  the 
R(f)  and  V(f)  for  each  subject’s  evaluation  measures.  As  the  various  w(v(x)) 

functions  are  not  standard  functions,  the  analysis  was  performed  numerically  in 
Microsoft®  Excel.  The  domain  was  divided  into  100  intervals  (except  for  the  ambush 
evaluation  measure,  which  was  treated  as  integer  units)  and  the  functions  assumed  to  be 
linear  over  the  interval  of  each  subdomain.  The  results  are  contained  in  Table  17.  Of  the 

80  u  (v(x))  functions  (four  for  each  of  the  20  subjects),  50  exhibited  risk  aversion,  15 

were  risk  neutral,  and  15  were  risk  seeking.  Clearly  the  common  assumption  that  risk 
aversion  is  present  does  not  hold  for  these  data  where  for  almost  two-fifths  of  the 
evaluation  measures  the  subjects  were  not  risk  averse.  A  two-way  analysis  of  variance 
provides  p  values  of  0.058  for  the  subjects  and  0.016  for  the  evaluation  measures.  The 
subjects  could  be  pooled  and  treated  as  a  group,  but  the  difference  of  at  least  one  subject 
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Table  17.  Tactical  Questionnaire  Results  for  R(f)  and  V(f). 


Artillery 

Ambush 

Fuel 

Time 

Subject  Number 

R(f) 

V(f) 

R(f) 

V(f) 

R(f) 

V(f) 

R(f) 

V(f) 

FJ1 

-0.142 

0.022 

0.102 

0.030 

0.380 

0.132 

0.260 

0.071 

FJ2 

0.143 

0.039 

0.939 

0.930 

0.373 

0.551 

-0.393 

0.187 

FJ3 

-0.043 

0.014 

0.205 

0.093 

0.353 

0.147 

0.438 

0.226 

FJ4 

-0.154 

0.030 

0.422 

0.204 

0.247 

0.058 

0.235 

0.134 

FJ5 

-0.281 

0.083 

0.413 

0.201 

0.113 

0.030 

0.091 

0.112 

FJ6 

0.373 

0.149 

0.767 

0.591 

-0.097 

0.022 

-0.276 

0.074 

FJ7 

-0.083 

0.036 

0.418 

0.182 

-0.441 

0.235 

-0.039 

0.052 

FJ8 

0.095 

0.028 

-0.035 

0.015 

0.310 

0.140 

-0.122 

0.057 

FJ9 

-0.089 

0.031 

0.530 

0.295 

-0.238 

0.075 

-0.127 

0.053 

FJ10 

0.138 

0.079 

0.239 

0.049 

0.031 

0.031 

0.080 

0.048 

FJ1 1 

-0.197 

0.069 

0.424 

0.206 

0.640 

0.425 

0.357 

0.170 

FJ12 

0.683 

0.470 

0.370 

0.133 

0.233 

0.137 

0.102 

0.163 

FJ13 

0.065 

0.065 

-0.059 

0.041 

-0.158 

0.043 

-0.074 

0.042 

FJ14 

0.016 

0.065 

0.568 

0.336 

0.175 

0.030 

0.228 

0.136 

FJ15 

0.684 

0.453 

0.604 

0.351 

0.142 

0.170 

0.606 

0.348 

FJ16 

0.314 

0.108 

0.599 

0.368 

0.077 

0.033 

0.204 

0.075 

FJ17 

0.447 

0.195 

0.357 

0.122 

0.773 

0.601 

0.474 

0.268 

FJ18 

-0.279 

0.088 

-0.594 

0.342 

0.436 

0.186 

-0.167 

0.040 

FJ19 

-0.147 

0.023 

0.655 

0.474 

0.633 

0.404 

0.550 

0.288 

FJ20 

0.388 

0.143 

0.526 

0.282 

0.251 

0.144 

0.377 

0.200 

approached  significance.  The  evaluation  measures  differed  significantly,  indicating  that 
the  subjects’  risk  attitudes  were  not  shared  according  to  evaluation  measure.  For 
example,  while  the  risk  attitudes  of  the  subjects  were  (barely)  similar,  no  conclusion  may 
be  drawn  that  they  were,  for  example,  risk  averse  to  a  certain  degree  regarding  the  enemy 
artillery  threat.  A  plot  of  R  (/)  versus  V  (/ )  appears  as  Figure  59. 

As  we  have  seen  that  knowledge  of  one  subject’s  risk  attitudes  cannot  provide 
information  on  another  subject’s  attitude  regarding  that  evaluation  measure  for  this  data 
set,  we  turn  to  examining  the  within  subject  risk  attitudes.  If  the  risk  attitudes  are  known 
for  a  number  of  evaluation  measures  for  a  specific  subject,  what  inferences  may  be  drawn 
about  the  subject’s  risk  attitudes  on  the  remaining  evaluation  measures? 
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-1.0  -0.5  0.0  0.5  1.0 


R(f) 

Figure  59.  Plot  of  R(f )  versus  V(f)  for  the  Tactical  Questionnaire.  Limit  shown  as 

solid  line. 

First  examining  this  issue  employing  R(f)  alone,  the  ranges  of  the  R  (/) , 
r  =  R*(f)-Ra(f),  where  R*  (/)  =  max{i?(/1),f?(/2),...,f?(/y)}  and 

R"  (/)  =  min  jf?  (f) ,  R(f2) R(  f. )} ,  for  the  subjects  of  the  tactical  questionnaire  are 

listed  in  Table  18.  As  the  ANOVA  results  permit  pooling  of  the  subjects’  data,  assume 
that  the  ranges  r  are  identically  independently  distributed.  Then  the  Central  Limit 
Theorem  provides  that  normality  is  present  for  sufficiently  large  samples.  Using  the 
tactical  questionnaire  sample  of  r,  with  20  samples,  it  is  detennined  that  r  is  distributed 
with  a  mean  of  0.63  and  a  standard  deviation  of  0.28.  Constructing  a  95  percent 
confidence  interval  for  such  a  distribution  provides  a  length  of  1 . 10. 

The  interpretation  of  this  calculation  is  that,  for  subjects  for  which  these  are 
representative,  presented  with  the  tactical  questionnaire,  once  three  utility  functions  have 
been  elicited,  the  fourth  lies  within  0.505  of  the  arithmetic  mean  of  the  known  R  (/) 
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Table  18.  Values  of  r ,  N ,  and  A  for  Tactical  Questionnaire  Subjects,  Ascending  Order. 


Subject 

r 

Subject 

N 

Subject 

A 

FJ10 

0.21 

FJ13 

0.10 

FJ13 

0.06 

FJ13 

0.22 

FJ10 

0.14 

FJ10 

0.07 

FJ20 

0.28 

FJ8 

0.15 

FJ20 

0.09 

FJ17 

0.42 

FJ1 

0.23 

FJ8 

0.15 

FJ8 

0.43 

FJ5 

0.25 

FJ3 

0.17 

FJ3 

0.48 

FJ9 

0.27 

FJ1 

0.18 

FJ1 

0.52 

FJ7 

0.28 

FJ4 

0.18 

FJ16 

0.52 

FJ3 

0.29 

FJ14 

0.19 

FJ15 

0.54 

FJ4 

0.29 

FJ16 

0.20 

FJ14 

0.55 

FJ14 

0.29 

FJ5 

0.20 

FJ4 

0.58 

FJ16 

0.33 

FJ17 

0.20 

FJ12 

0.58 

FJ18 

0.41 

FJ15 

0.20 

FJ5 

0.69 

FJ12 

0.42 

FJ12 

0.23 

FJ9 

0.77 

FJ20 

0.43 

FJ7 

0.26 

FJ19 

0.80 

FJ6 

0.44 

FJ9 

0.27 

FJ1 1 

0.84 

FJ1 1 

0.46 

FJ1 1 

0.28 

FJ7 

0.86 

FJ19 

0.58 

FJ19 

0.32 

FJ18 

1.03 

FJ17 

0.60 

FJ18 

0.33 

FJ6 

1.04 

FJ15 

0.61 

FJ6 

0.43 

FJ2 

1.33 

FJ2 

0.64 

FJ2 

0.53 

values  with  a  probability  of  0.95.  The  extreme  length  of  this  confidence  interval, 
extending  across  almost  one-half  of  the  domain  space,  renders  it  rather  useless. 

An  improved  analysis  would  include  V(f).  Two  approaches  are  considered,  one 


with  respect  to  the  origin  and  the  other  with  respect  to  the  average  R(f)  and  V (/)  for  a 

subject.  The  former  is  a  measure  of  the  distance  from  overall  risk  neutrality  for  the 
subject.  As  we  are  working  in  two-dimensional  Cartesian  space,  the  total  distance  from 
the  origin,  N  ,  is  straightforward  and  is  defined 


N  = 


J 


iO.S 


The  distance  from  the  average  R(f.) 


\£r(/,)  a»d  v{/.) 

J  7=1 


(90) 


determined  utilizing 
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(91) 


a=}s|"(^(/j)-^(/-))2+(’,(/,)-’,(/.))2T 

J  j=il  J 

The  values  of  N  and  A  are  presented  in  Table  18  in  ascending  order.  Inspection 
shows  that  these  measures  differ  from  each  other  as  well  as  r.  As  measures,  N  and  A 
provide  more  information  and  so  are  more  appealing.  Similarly  as  was  done  for  r,  it  is 
determined  that  A  is  distributed  with  a  mean  of  0.23  and  a  standard  deviation  of  0. 1 1, 
producing  a  95  percent  confidence  interval  of  length  0.44.  A  method  of  utilizing  this 
information  would  be  for  an  individual  sharing  the  characteristics  of  this  group  of 
subjects,  elicit  utility  functions  for  three  of  the  evaluation  measures.  The  fourth  utility 
function  will  possess  R  (/)  and  V(f)  within  a  radius  0.22  of  the  mean  of  the  three 

known  functions  with  a  probability  of  95  percent.  Unfortunately,  this  radius  is  also  rather 
large,  making  this  approach  inoperable  as  a  useful  approach. 

The  idea  that  subjects  are  risk  attitude  consistent  is  not  supported  by  the  tactical 
questionnaire  data.  Thirteen  of  the  20  subjects  exhibit  R  (/)  that  change  sign. 

Removing  the  two  subjects  for  whom  the  sign  changes  take  place  within  the  risk  neutral 
region,  eleven  subjects  were  risk  seeking  for  at  least  one  evaluation  measure  and  risk 
averse  on  at  least  one  other. 

It  is  possible  that  future  work  with  larger  sample  sizes  and  different  decision 
situations  could  produce  a  confidence  interval  that  is  diminutive  enough  to  be  practical. 
Barring  such  improvements,  the  idea  that  a  decision  maker’s  known  risk  attitudes  may 
provide  insight  into  risk  attitudes  on  other  evaluation  measures  is  not  operational.  The 
data  of  this  study  suggest  that  the  idea  is  not  sound,  but  rather  that  risk  attitudes  are 
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completely  independent  for  different  evaluation  measures  for  a  single  individual, 
particularly  with  respect  to  risk  aversion  and  risk  seeking  behavior. 

We  conclude  this  section  by  observing  that  R(f )  and  V (/ )  provide  improved 

an  improved  metric  for  evaluating  risk  attitudes  over  both  curve  shape  and  preference 
area.  Metrics  for  the  assessment  of  risk  attitude  consistency  are  developed:  r,  N  and  A  . 
Risk  attitude  consistency  is  not  demonstrated  by  the  tactical  questionnaire  data.  Instead, 
the  research  subjects  exhibit  wide  differences  in  risk  attitude  by  evaluation  measure. 


Conclusions 

This  research  supports  the  hypothesis  that  single  dimensional  value  and  utility 
functions  are  not  equivalent.  This  conclusion  was  confirmed  for  two  separate  military 
decision  making  situations.  No  single  functional  relationship  defined  the  relationship  of 
value  and  utility  functions.  Certainty  equivalent  and  probability  equivalent  and 
Multiattribute  elicitation  methodologies  provided  equivalent  results.  Value  and  utility 
functions  differed  significantly  more  than  can  be  explained  by  elicitation  error. 

A  new  metric  for  assessing  preference  function  fit,  WRMSE,  was  developed.  The 
frequent  assumption  of  risk  aversion  was  a  poor  one  for  this  subject  group.  Risk  attitudes 
were  not  found  to  be  consistent  between  evaluation  measures  for  the  subjects.  New 
metrics  for  assessing  risk  attitudes  were  presented. 
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V.  Hybrid  Value-Utility  Decision  Analysis  Models 


General 

As  discussed  previously,  decision  analysis  models  may  be  utility-function-based, 
or  value-function-based.  Research  in  Chapter  IV  showed  that  the  two  models  differ 
significantly.  The  utility-based  model  is  appropriate  when  outcomes  are  not  certain,  as  is 
typical  for  difficult  decisions.  As  utility  functions  incorporate  the  risk  attitudes  of  an 
individual,  they  are  specific  to  an  individual.  Therefore  construction  of  utility  models 
requires  a  significant  amount  of  the  decision  maker’s  time.  For  large  decision  problems, 
this  personal  commitment  on  the  part  of  the  decision  maker  is  onerous  and  is  a  barrier  to 
the  use  of  multiple  objective  decision  analysis.  Value -based  models,  appropriate  under 
conditions  of  certainty,  may  be  constructed  through  consultation  with  appropriate  experts 
rather  than  the  decision  maker.  In  fact,  consultation  with  subject  matter  experts  is 
generally  required  in  any  complex  problem,  regardless  of  methodology  applied. 

If  a  utility-based  model  may  be  approximated  by  modification  of  a  value-based 
model,  reduction  of  the  time  demands  on  the  decision  maker  may  be  realized.  For  large, 
important  decisions,  these  time  demands  are  significant.  Reducing  the  time  required  of 
the  decision  maker  improves  the  acceptance  of  the  procedure  and  improves  the 
probability  that  decision  analysis  will  be  employed.  The  objective  of  this  portion  of  the 
research  is  to  detennine  when  single  dimensional  value  and  utility  functions  differ 
significantly,  and  use  this  infonnation  to  construct  a  hybrid  value-utility  model  that  may 
be  used  to  estimate  the  performance  of  the  utility  model. 


148 


This  chapter  will  first  review  pertinent  background  information.  An  algorithm 


will  be  presented  for  construction  of  the  hybrid  model.  The  algorithm  will  be  tested  on  a 
simple  example  problem.  Next,  the  algorithm  will  be  used  to  extend  work  preformed  for 
a  client  to  select  an  automatic  target  recognition  classification  system.  Finally  the  chapter 
is  summarized  and  potential  future  work  presented. 


Background 

Decision  analysis  models  consist  of  a  set  of  evaluation  criteria  for  which 
preference  functions  are  used  to  assess  the  decision  maker’s  strength  of  preference  for 
various  levels  of  the  criteria,  and  often  also  attitudes  regarding  the  uncertainty  of  the 
decision  situation.  In  a  typical  value-focused  approach  (Keeney,  1992a),  the  evaluation 
criteria  are  determined  through  decomposition  of  the  decision  maker’s  strategic  priorities 
until  measurable  criteria,  the  evaluation  measures,  are  obtained.  This  inverted  dendritic 
structure,  the  value  hierarchy,  is  shown  in  the  upper  portion  of  Figure  60.  Each 
evaluation  measure  is  assessed  using  a  single  dimensional  preference  function.  Under 
conditions  of  certainty,  the  preference  function  is  referred  to  as  a  value  function  and 
measures  the  decision  maker’s  strength  of  preference.  Under  uncertainty,  the  preference 
function  is  referred  to  as  a  utility  function  and  measures  both  the  decision  maker’s 
strength  of  preference  and  her  attitude  toward  the  risk  inherent  in  the  uncertain  decision 
context.  The  preference  functions  are  contained  in  the  second  from  bottom  row  of  Figure 
60.  The  various  possible  levels  of  the  evaluation  measure  variables  are  represented  as  the 
bottom  row  of  Figure  60.  A  multiattribute  (multidimensional)  function  employing 
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relative  weights  (relative  importance)  of  the  various  evaluation  measures  combines  the 
single  dimensional  preference  functions  and  completes  the  model,  permitting  evaluation 
of  alternatives.  Figure  60  represents  a  common  view  of  decision  analysis  models,  which 
assumes  perfect  information.  (In  fairness,  it  must  be  observed  that  sensitivity  analysis  is 
often  used  to  examine  the  assumption  of  perfect  information.) 


IA  \A  Lk  LA  ...  LA 

,  4  ,  o  ,  t  ,  a  ,  s  ,  ii  i 

IAIzulA_  IZulzulA_  ...  hi 


Figure  60.  Common  Decision  Analysis  Model  Structure.  The  bottom  tier  represents  the 
evaluation  measures  and  the  next  tier  the  preference  functions.  The  hierarchy  represents 
the  decomposed  values  of  the  decision  maker. 

Decision  analysis  models  are  actually  more  complex.  This  complexity  is  often 
largely  ignored  for  tractability.  Figure  61  shows  the  taxonomy  of  variation  in  a  DA 
model.  The  bottom  tier  represents  the  evaluation  measure  levels.  These  levels  may  be 
fixed,  under  certainty  when  the  evaluation  measure  levels  are  known,  or  stochastic,  under 
uncertainty  or  risk.  Under  the  latter  condition  the  evaluation  measure  levels  are 
distributed  in  a  known  or  unknown  way,  respectively  (usually  referred  to  as  under  risk  or 
uncertainty).  Additionally,  the  evaluation  measure  levels  are  often  a  function  of  an 
alterative,  with  different  alternatives  possessing  differing  distributions.  These 
distributions  make  the  utility  model  a  random  variable. 
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Only  one  distribution  is  shown  for  each  evaluation  measure  in  the  figure.  The 
figure  shows  continuous  distributions,  but  discrete  distributions  and  simple  scalars  (no 
variation)  are  possible.  The  next  higher  tier  represents  the  estimates  of  the  true 
distributions.  The  third  tier  from  the  bottom  illustrates  the  value  functions.  Above  them 
are  the  value  probability  density  function  distributions  created  by  transforming  the 
evaluation  measure  distributions  by  the  value  functions.  Above  the  value  distributions 
are  the  utility  functions.  Two  forms  of  utility  functions  are  employed.  The  domain  of 
one  form  is  the  evaluation  measure  level  domain.  The  domain  of  the  other  is  the  value 
function  codomain.  Depending  which  fonn  is  used,  differing  prior  information  is 
required,  as  indicated  by  the  arrows  on  the  right.  The  utility  functions  provide  utility 
probability  density  function  distributions  when  the  inherent  uncertainty  derived  from  the 
evaluation  measures  is  transformed.  Note  that  both  value  and  utility  functions  produce  an 
associated  preference  distribution.  Each  evaluation  measure  has  an  associated  elicited 
weight.  Elicitation  of  these  weights  involves  elicitation  error,  so  that  the  weights,  wt ,  are 

associated  with  some  distribution  and  represented  by  an  estimate,  wt .  For  our  purposes, 
in  this  document  such  error  is  disregarded,  and  vv,  =  wt .  Notation  is  indicated  on  the  right 
hand  side  of  Figure  6 1 . 

Finally,  a  function  is  employed  to  aggregate  the  decision  maker’s  strength  of 
preference  and  risk  attitudes  across  the  various  dimensions  of  the  decision.  A 
multiattribute  function  for  a  utility  model  is  often  written  in  the  additive  form.  This  is 

U;  =  iX  "'."A)]  («) 

1=1 
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where  U .  is  the  utility  of  the  /th  alternative,  wt  is  the  weight  (relative  importance)  of  the 
z'th  evaluation  measure,  and  w,  (x, )  is  the  single  dimensional  utility  function  for  the  /'th 
evaluation  measure,  x,  is  the  random  variable  for  the  z'th  evaluation  measure,  and  /  is  the 
number  of  evaluation  measures. 

Equation  (92)  may  be  rewritten  in  tenns  of  the  utility  probability  density  function, 
or 
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(93) 


tlwifi"(ui(xi),ay) 


i= 1 


i=i 


where  f"  is  the  probability  density  function  of  the  zth  utility  function,  and  d(  is  a  vector 
of  parameters  of  the  estimated  evaluation  measure  distributions.  The  DA  problem  may 
be  stated  as  maximizing  E  yj  j  J  through  the  selection  of  the  /th  alternative.  More 


formally,  we  take  the  maximum 

Max  E[Uj~\ 


(94) 


such  that 


E[Uj]  =  E 


(95) 

(96) 


/  =  1,...,/  (97) 

7  =  1,...,/  (98) 


Figure  62  illustrates  an  example  where  there  are  two  alternatives,  A  and  B. 
Choosing  A  or  B  is  the  only  decision  and  is  followed  by  a  differing  uncertain  event. 
These  events  have  distinct  distributions  of  overall  utility  associated  with  it.  In  Figure  62, 
choosing  B  will  provide  a  higher  expected  value  of  utility.  Or  to  avoid  the  confusion 
associated  with  the  statistical  term  “expected  value,”  one  may  say  choosing  B  will 
provide  a  higher  mathematical  expectation  of  utility.  More  often,  this  is  shortened  to 
stating  that  choosing  B  will  provide  a  higher  expected  utility. 
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Figure  62.  Example  Decision  Tree.  Squares  represent  decisions,  circles  represent 
discretized  uncertain  events.  The  utility  PDF  for  each  choice  is  shown  at  right.  These 
distributions  are  typically  discretized  for  representation  in  decision  analysis. 


For  a  value -based  model,  the  additive  multiattribute  function  is 

»0  =  2>,v,(*,)  (99) 

i= 1 

where  F.is  the  value  of  the  /th  alternative  and  vi  (x;)  is  the  single  dimensional  value 

function  for  the  z'th  evaluation  measure.  Equation  (99)  contains  no  expectation  operator, 
as  it  is  intended  for  us  under  conditions  of  certainty.  However,  it  may  be  seen  from 
Figure  61  that  using  value  functions  under  uncertainty  produces  a  value  pdf  that  behaves 
similarly  to  the  utility  pdf.  Equation  (99)  then  becomes 

r,  =  'fcE[w,(*i)'\  (100) 

1=1 

In  analogous  fashion,  an  objective  value  function  may  be  constructed 

Ma  xf[f;]  (101) 

such  that 
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(102) 

II 

_  /=i  J 

(103) 

(104) 

(105) 

where  f'  is  the  probability  density  function  of  the  zth  value  function,  and  a'  is  a  vector  of 
parameters  of  the  estimated  evaluation  measure  distributions.  The  DA  problem  may  be 
stated  as  maximizing  £'[F/.]  through  the  selection  of  the  /th  alternative. 

In  uncertain  decision  contexts  Equation  (93)  is  appropriate  but  for  large  problems 
of  great  import  the  ui  (x; )  are  difficult  to  elicit  from  the  decision  maker  because  of  the 

time  involved.  As  risk  attitudes  vary  within  an  organization,  it  is  not  acceptable  to  use  a 
surrogate  for  the  decision  maker  to  elicit  utility  functions.  Arguably,  an  organization 
should  have  a  consensus  for  values  in  a  decision  situation,  so  parameters  for  Equation 
(99)  are  may  be  elicited  from  subject  matter  experts.  Modifying  Equation  (99)  provides 
an  approach  to  estimate  Equation  (92)  to  within  some  specified  accuracy.  The  advantage 
of  employing  a  different  model  that  adequately  approximates  Equation  (92),  but  is  based 
on  Equation  (99),  is  the  reduction  of  the  elicitation  burden  on  the  decision  maker.  This 
onerous  burden  sometimes  either  precludes  employing  a  utility  approach  or  causes  the 
selection  of  non-decision  analysis  techniques,  and  has  been  identified  as  a  limitation  of 

this  methodology.  It  is  possible  to  do  so  as  both  f'  and  /  map  from  the  real  numbers 
into  the  unit  interval,  as  was  presented  above  in  the  discussion  of  Figure  61. 
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The  proposed  procedure  is  depicted  graphically  in  Figure  63.  Starting  with  the 
value  model,  a  hybrid  model  is  created  by  replacing  certain  single  dimension  value 
functions  with  the  corresponding  utility  functions.  The  resultant  hybrid  model  is  then 
used  as  would  the  utility  model  to  analyze  the  decision  situation. 


Figure  63.  Basic  Hybrid  Value-Utility  Model  Approach. 


Typically,  the  initial  step  of  building  the  value  model  is  done  through  interview  of 
the  decision  maker.  As  we  wish  to  minimize  the  time  required  of  this  individual,  we  will 
construct  the  value  model  using  appropriate  subject  matter  experts.  The  decision  maker 
then  should  validate  the  value  model.  As  we  are  concerned  with  complex  decision 
situations,  often  we  would  expect  the  number  of  evaluation  measures  to  be  large.  Group 
screening  is  a  method  that  efficiently  examines  large  numbers  of  variables.  We  anticipate 
that  group  screening  will  be  employed  as  part  of  the  step  detennining  significance  of  the 
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evaluation  measures.  Further  development  of  the  methodology  outlined  in  Figure  63  is 
provided  below. 


A  Hybrid  Value-Utility  Algorithm 


Algorithm  Development.  A  method  of  approximating  Equation  (92)  is  desired. 
Beginning  with  Equation  (99),  one  evaluation  measure’s  value  function,  vk  {xk ) ,  is 

replaced  with  the  appropriate  utility  function,  uk  (xk ) .  The  model  becomes 


i 

=  E 

Z  wif'(vi  (xi )’«&  )  +  wJk(uk  ixk  )’K) 


i=l,i*k 


(106) 


where  the  Mi  evaluation  measure  value  function  has  been  replaced  with  the 
corresponding  utility  function  and  Uj  is  an  estimate  of  U .  after  one  substitution. 
Equation  (106)  then  becomes,  after  m  substitutions, 


U 


(■ m ) 


Z[(i-*,)  wif'(vi  (a)’  ) + Ktw{f'(  ui  ( x{ ) ,  a;;  ) 


(107) 


L  i= 1  J 

where  K.t  is  an  indicator  variable  corresponding  to  the  substitutions  defined  for  the  zth 
evaluation  measure  as 


f  0  when  vf  ( xt )  remains, 

1  1  when  ui  ( x{ )  substituted 


(108) 


This  substitution  protocol  continues  in  an  iterative  fashion  until  the  estimated  utility, 


U\1"'1  is  sufficiently  close  to  U  j .  This  may  be  stated  as 


- ElUJ ] 


<  8 ,  where 
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8  is  the  level  of  accuracy  desired.  Then  E 


U' 


(m) 


ElUj]- 


Equation  (107)  is  a  hybrid 


value-utility  model  that  estimates  the  true  utility  model  behavior. 

Typically  decision  analytic  studies  proceed  using  either  a  value  or  a  utility  based 
approach.  Figure  64  illustrates  that  after  the  evaluation  measure  variation  is  estimated, 
the  analyst  adopts  a  preference  model  of  value  or  utility.  Using  the  selected  preference 
function  for  each  evaluation  measure,  the  evaluation  measure  scores  are  transfonned  into 
the  value  or  utility  space.  Typically  these  evaluation  measures  have  uncertainty 
associated  with  them,  so  the  various  levels  of  value  and  utility  have  some  associated 
probability.  Figure  64  shows  probability  density  functions  for  value  and  utility  levels. 

As  shown  at  the  top  of  Figure  64,  Equation  (107)  bridges  the  division  between  value  and 
utility  regions.  In  all  cases  in  Figure  64,  an  additive  value  model  is  shown  as  an  example. 

Substitution  order.  Two  additional  considerations  must  be  addressed  in  order  to 
develop  an  algorithm.  These  are  the  order  of  the  substitutions  of  single  dimensional 
utility  for  single  dimensional  value  functions,  and  when  the  substitutions  may  be  halted. 
The  interview  session  time  cost  of  each  substitution  (each  evaluation  measure)  is  roughly 
equal  because  the  procedure  is  identical.  Some  evaluation  measures  may  require  more 
reflection  or  definition,  and  so  require  longer  elicitation  periods.  This  time  cost  is  not 
considered  in  this  analysis.  With  this  simplification,  the  substitutions  should  be  made  in 
order  of  decreasing  impact  upon  the  hybrid  model  (107).  The  substitutions  may  be 
stopped  when  sufficient  accuracy  is  achieved,  as  discussed  below. 

Several  candidates  exist  for  prioritizing  the  substitutions.  The  relative  weight,  vv\  , 
is  an  obvious  possibility.  However  the  contribution  of  each  evaluation  measure  is  the 
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Figure  64.  Bifurcation  of  Preference  Functions. 


product  of  the  weight  multiplied  by  the  corresponding  preference  function.  Evaluation 
measures  with  high  relative  importance  may  not  be  significant  contributors  to  the  overall 
value  or  utility.  That  is,  an  ordering  established  by  wt  may  be  very  different  than  an 

ordering  based  on  wivi  (xt ) .  While  the  product  £[vvv  (x,  )]  may  be  calculated  for  each 
evaluation  measure  for  the  alternatives,  this  is  not  a  good  predictor  of  E  [ wjui  (x, )] .  As 
was  shown  in  the  last  chapter,  generally  vf  (xi )  ^  m.  (xj ,  so  value  is  a  poor  predictor  of 
utility. 

While  assumptions  like  consistent  risk  attitude  across  evaluation  measures  may  be 
used  to  improve  the  estimate  of  E\wiui  (x; )] ,  response  surface  methodology  (RSM) 

provides  a  means  to  assess  the  potential  relative  contributions  of  E\wiui  (x; )]  without 
introducing  additional  constraints  of  the  decision  maker’s  risk  attitudes.  The  coefficients 
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of  the  regression  model  produced  by  RSM  indicate  the  potential  contribution  of  each 
model  variable  and  pennit  the  prioritization  of  substitution  of  the  evaluation  measures. 

The  single  dimensional  preference  functions  transfonn  the  inherent  problem 
variation  into  value  and  utility  probability  density  functions.  While  usually  not  directly 
considered,  these  derived  PDFs  affect  the  decision  analysis  model.  Differing  single 
dimensional  utility  functions  produce  differing  utility  PDFs,  as  shown  in  Figure  65. 

Frequently,  utility  functions  are  approximated  employing  parametric  functions 
fitted  to  the  data  (Kirkwood,  1997:  65).  When  the  exponential  function, 

~V‘{X% 

¥/(fU))=  1  g  -i/  (109) 

1  -  e  /p‘ 

is  used  to  approximate  a  utility  function  the  exponential  constant,  p ,  may  be  modified  to 
examine  an  envelope  of  utility  curves.  Setting  /?,  =°°  for  ui  (v(x;))  establishes  that 

there  are  no  risk  attitudes  and  therefore  ut  (v(x,))  =  v;  (x;) .  Varying  pt  permits 

employment  of  response  surface  methodology  to  detennine  whether  each  ui  (x;.) 

potentially  exerts  a  significant  effect  on  the  hybrid  model.  Kirkwood  (1997b:  66) 
indicates  that  the  absolute  value  of  p  is  typically  greater  than  one-tenth  of  the  evaluation 

,  .  max  X:  -  min  x, 

measure  domain,  or  pt  > - - -  .  The  region  bounded  this  relationship  is 

illustrated  in  Figure  66.  Accepting  the  limits  proposed  by  Kirkwood  means  that  if  we 
examine  the  two  ul  (x, )  for  p  =  ±0. 1 ,  and  they  do  not  significantly  affect  the  hybrid 
model,  then  the  preference  function  for  the  i  th  evaluation  measure  may  remain  a  value 
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Figure  65.  Affect  of  Various  Utility  Functions  of  the  Utility  PDF. 

function.  We  do  not  expect  the  utility  function  for  that  evaluation  measure  to 
significantly  affect  the  hybrid  model. 

The  limits  proposed  by  Kirkwood,  coupled  with  RSM,  provide  an  excellent  way 
to  assess  the  potential  significance  of  the  utility  functions.  Therefore  RSM  is  a  desirable 
methodology  to  use  to  order  the  substitution  of  the  evaluation  measures  for  the  hybrid 
utility  model  algorithm. 
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V,U) 

Figure  66.  Exponential  Utility  Function  Limits.  The  straight  line,  where  uj  (xi )  =  v.  (x;. ) , 
occurs  when  p  =  °°  .  The  other  curves  show  ut  (x(. )  when  p  =  ±0. 1 . 


Stopping  Criteria.  Replacing  all  potentially  significant  evaluation  measures  will 
ensure  that  the  hybrid  model  adequately  represents  the  utility  model.  However,  it  is 

desired  to  halt  the  iterative  substitutions  when  E  U{'" 1  -  E^U^  <  8 ,  which  may  occur 

before  all  the  evaluation  measures  have  been  converted.  We  may  write  the  problem  as  a 

mathematical  programming  problem.  The  optimization  model  is: 

/ 

min  i»=2X  (no) 

i= 1 

subject  to 

E\u\m)~\-E[Uj]  <S\/j  (111) 

[0,1]  (112) 

However,  £’[U/]  is  unknown,  so  it  is  not  directly  possible  to  determine  when  E  Uj") 
is  within  8  of  is  [le]  .  When  there  is  consistent  risk  aversion  (risk  affinity)  then  the  set 


162 


of  all  possible  E  I/jm)  ,  SJ ,  is  bounded  below  (above)  by  F.  and  above  (below)  by 

.  See  Figure  67  for  examples  of  behavior  of  preference  function  under  various 
risk  attitudes.  When  risk  aversion  is  present  for  the  zth  evaluation  measure, 
ui  (v,  (x,))  >  v;.  (x.)  .  The  reverse  is  found  under  risk  seeking  behavior.  Mixed  risk 

attitudes,  as  typically  encountered  in  this  research  (see  Chapter  IV),  preclude  such 
general  conclusions.  Figure  68  shows  iterative  ordered  substitutions  of  decreasing  impact 
under  conditions  of  risk  aversion  and  risk  seeking  behavior. 


Risk  Averse  Risk  Seeking  Mixed  Risk  Attitude 

Figure  67.  Risk  Attitudinal  Effects  on  the  Relationship  of  Value  and  Utility 

Functions. 


Definition  (Substitution  Distance).  Given  a  series  of  substitutions  to  create 
hybrid  value-utility  models  for  the yth  alternative,  S/  -  jk. ,  uf* ,  U{p , . . . ,  U'"')| ,  the 

distance  between  successive  substitutions  is  given  by  A1"1'1 ,  defined  as 

Ajm)  =  e\  ]  -  e\ ]  (113) 

where  m  indicates  the  number  of  substitutions,  and  E  U'0)  =  E'[F/-]  • 
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Figure  68.  Iterative  Decreasing  Ordered  Substitutions.  Shown  are  the 
substitutions  for  a  single  alternative,  j. 


When  the  set  S  ■  is  ordered  with  decreasing  intervals  between  iterative  estimates, 


^(m)  <  l)ymG  {2,3,...,/} ,  then  S,  is  a  partially  ordered  set.  Under  this  condition, 


when  (/  - m)X{'“]  <  8 ,  then 


U' 


( m ) 


-E[Uj]<8.  When  (/  -  m)  /L;(m)  <  S  is 


encountered  the  substitutions  of  single  dimensional  utility  functions  for  value  functions  is 


halted  and  the  current  E 


U 


(m) 


accepted  as  accurately  estimating  E  U .  . 


There  are  a  number  of  potential  alternatives  of  interest,  not  just  alternative  j. 
Define  the  vectors  U  and  Cl1"'* 


~u; 

u  = 

u/\ 

and  Ul"!)  = 

JjH 

Uj\ 

u?_ 

(114) 
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Clearly  it  is  not  necessary  to  continue  iterations,  and  so  to  increment  m,  when 
E  U'"0  -£[U,]  <  8  V/  .  What  is  of  interest  is,  naturally,  the  alternative  with  the 

greatest  utility.  Therefore  we  are  satisfied  when  the  alternative  with  the  greatest  utility  is 
alone  within  the  accepted  tolerance.  Define  0im]  =  max|ci("'*,t/2",),---,C(/m)|  where  m  is 

the  most  recent  iteration.  Define  =  E  0i'")  -E  u[m~v>  where  *  represents  the 

evaluation  measure  with  the  greatest  utility  in  the  last,  mth,  iteration.  The  mathematical 
programming  formulation  now  becomes: 

/ 

min  m  =  ZK,  (H5) 

i= 1 

subject  to 

(/ -  m)  <  8  (116) 

Sg[0,1\  (117) 

When  Sj  is  partially  ordered,  m  is  minimized  when  equation  (1 16)  is  first  found  to  hold. 
Stop  the  algorithm  when  (/  -  m)Ai"'}  <  8  and  accept  E  U\"l)  as  accurately  estimating 

C[l7*]  and  the  alternative  corresponding  to  u[m)  as  being  the  optimal  choice. 

Group  Screening.  As  stated  previously,  the  class  of  decision  analysis  problems 
considered  on  which  these  techniques  are  likely  to  be  beneficial  are  those  with  a  large 
number  of  evaluation  measures.  For  such  problems,  group  screening  is  likely  to  be 
required.  The  group  screening  employment  is  as  traditionally  employed  as  presented  in 
Chapter  II.  Once  the  screening  is  completed,  the  information  must  be  re-aggregated, 
which  is  a  new  contribution. 
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Employing  groups  screening  permits  examination  of  a  large  number  of  variables 
through  decomposing  the  independent  variables  into  subsets.  Once  the  groups  screening 
is  completed,  it  is  desired  to  reassemble  the  variables  into  a  single  set,  rank  ordered  by 
the  degree  of  affect  each  has  on  the  model.  The  utility  model  is 

U  =  YJwiui(vi(xi))  (118) 

i=i 

where,  with  the  assumption  of  the  exponential  single  dimensional  model,  the  ith 
evaluation  measure  is  determined  by 

-v,(0/ 

¥,(eU))=  I  g  -1/  (119) 

l- e  /pi 

When  the  Mi  element  of  (1 18)  is  examined  separately,  the  model  becomes 

k- 1  n 

U  =  Z  w&  (vi  ( A )) +  wPh  ('4  (-A ))+  Z  cixi  (] 12°) 

i= 1  i—k+l 

As  equation  (118)  may  be  represented  by  a  fit  of  a  linear  model  based  on  the  exponential 
constant 


U  =  c0  +  ZC«A  (121) 

i= 1 

the  Mi  element  of  (120)  may  be  decomposed  from  (121)  and  modeled  by  a  least  squares 
model 


k- 1 


=c0+YjciPi+ckPk  +  X  ca 


(122) 


1=1 


i=k+ 1 


The  coefficient  ck  provides  a  measure  of  importance  of  pk  relative  to  the  other  pt . 
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If  the  Mi  element  has  been  selected  to  be  treated  as  a  group  of  variables  for 
screening,  then  it  becomes  necessary  to  decompose  the  Mi  group  when  it  is  determined  to 
be  significant.  Because  the  Mi  group  may  also  be  modeled  by 

n 

Uk  =  Ck0  +  TcuPu  (123) 

i= 1 

the  contribution  to  the  overall  model  for  the  pkm  th  variable  is  ckckm  .  Similar 

combinations  are  made  for  all  grouped  variable  coefficients  to  permit  comparison  of 
evaluation  measure  priorities.  The  relative  importance  of  utility  functions  for  the 
evaluation  measures  is  provided  by  the  corresponding  ckckm  •  •  •  ck{r_V]ckr  where  r  is  the 

number  of  screening  sets  in  which  the  kt h  evaluation  measure  has  been  grouped.  Rank 
ordering  the  evaluation  measures  by  relative  importance  provides  the  order  for  the 
substitutions  from  single  dimensional  value  function  to  single  dimensional  utility 
functions  when  the  hybrid  model  is  created. 

The  Detailed  Algorithm.  Refining  the  basic  concept  provided  in  Figure  63  by 
minimizing  the  interactions  required  with  the  decision  maker,  incorporating  group 
screening,  and  using  response  surface  methodology  to  assess  potential  significance 
provides  the  algorithm  presented  in  Figure  69. 

The  steps  of  this  protocol  are: 

1 .  Build  Value  Model. 

a.  Establish  the  multiattribute  functional  form.  Typically  the  simple 
additive  fonn  is  used,  but  the  algorithm  is  independent  of  the 
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Build  Value  Model 


Establish  Multiattribute  Functional  Form 


Figure  69.  Steps  to  Create  Hybrid  Value -Utility  Model. 
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multiattribute  functional  form.  A  constraint  on  the  algorithm  is 
that  the  value  model  and  the  utility  model  must  be  of  the  same 
fonn. 

b.  Elicit  value  hierarchy  and  evaluation  measure  weights,  wl ,  from 

subject  matter  experts.  This  avoids  a  significant  time  burden  being 
placed  upon  the  decision  maker. 

c.  Verify  the  value  hierarchy  and  evaluation  measure  weights  with 
decision  maker.  Clearly  the  model  must  be  acceptable  to  the 
decision  maker.  Verification  at  this  point  provides  efficient 
elicitation  of  the  single  dimensional  value  functions. 

d.  Elicit  v(.  (x.)  for  all  i  from  subject  matter  experts. 

2.  Determine  potential  impact  of  single  dimensional  utility  functions. 

a.  Group  vf  (x, ) .  Grouping  is  done  to  permit  group  screening,  which 

allows  examination  of  large  numbers  of  evaluation  measures.  The 
value  hierarchy  provides  a  ready  framework  for  grouping  of 
variables.  An  obvious  approach  is  to  group  variables  at  the  highest 
partitioning.  Each  group  may  be  subsequently  examined, 
employing  group  screening  as  required.  An  additional 
consideration  is  that  if  any  interactions  are  believed  to  be  present, 
the  groupings  should  not  separate  those  variables. 

b.  Develop  and  execute  experimental  design.  Design  of  experiments 
pennits  the  significance  of  the  predictor  variables  to  be  assessed 


169 


with  the  minimum  number  of  experimental  runs.  A  run  in  this 
sense,  is  setting  the  variables  at  some  level  and  observing  the 
expected  utilities  of  the  various  alternatives.  The  variables  of 
interest  with  respect  to  the  hybrid  model  are  the  exponential 

constants  of  the  ui .  (v,.  (jc,-  ))  functions.  As  the  exponential 
constants  assume  values  of  p  =  {-0. 1,  °°,  0. 1} ,  the  fractional 

factorial  with  center  point  designs  are  attractive.  The  experimental 
design  is  complicated  by  the  consideration  of  the  set  of 
alternatives.  The  alternative  choice  may  be  included  as  a  variable 
in  the  experimental  design.  If  the  set  is  large,  the  number  of 
available  experimental  designs  is  small.  This  step  and  the 
following  step  are  repeated  for  each  grouping  of  variables, 
c.  Employ  RSM.  Examine  potential  contribution  of  uf  (xi )  in  the 

current  model  in  place  of  v(  (x,. )  by  employing  response  surface 

methodology  to  examine  ui  (v;.  (x;)) .  This  analysis  provides 

regression  coefficients  showing  the  contribution  of  each  variable  as 
well  as  significance  information.  This  provides  data  that  facilitates 
detennining  the  evaluation  measures  that  need  to  be  considered  for 
conversion  into  utility,  at  some  defined  alpha  level,  as  well  as 
straightforward  prioritization  information.  This  step  is  done  for 
each  grouping  of  variables. 
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d.  Ungroup  vi  (x; ) .  The  RSM  information  from  the  previous  step  is 

ungrouped  to  provide  a  common  comparison  for  all  evaluation 
measures.  Linear  regression  coefficients  are  multiplied  by  the 
coefficients  for  all  parent  groups  to  provide  a  single  rank-ordered 
list  of  the  potentially  significant  evaluation  measures. 

3.  Replace  potentially  significant  evaluation  measures  with  utility  functions. 
In  priority  order,  perform  iterative  substitutions.  This  is  done  by: 

a.  Set  iteration  counter  m  =  1 

b.  Select  the  remaining  v;.  (x, )  from  the  list  of  potentially  significant 

evaluation  measures  with  the  greatest  coefficients  from  the  RSM 
model. 

c.  Elicit  uj  (x, )  for  this  evaluation  measure. 

d.  Determine  U{’n)  for  all  j  by  substituting  the  m(  (x,.)  from  the 
previous  step. 

e.  If  m  =  1  or  if  either  stopping  criteria  is  not  met,  increment 
m  =  m  + 1  and  return  to  step  b.  Otherwise  stop  and 

E  0l'n}  ~  E  [£/,] .  The  stopping  criteria  are: 

i.  All  significant  evaluation  measures,  as  detennined  by 
RSM,  have  been  converted  from  vt  (x,.)  to  w.  (x;) . 

ii.  E  O'"1'1  is  sufficiently  close  to  E^U ^ . 

4.  The  hybrid  model  is  now  used  as  would  a  standard  utility  model  to: 
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a.  Select  the  best  alternative,  ,  from  an  expected  value 
standpoint,  and 

b.  Perfonn  a  sensitivity  analysis. 


Example 

Problem  Statement.  This  example  provides  a  simple  situation  with  two 
alternatives  and  two  evaluation  measures.  Consider  a  project  where  a  parking  lot  must  be 
constructed.  As  a  lot  must  be  constructed  for  some  reason,  the  alternative  of  the  status 
quo  is  not  acceptable.  The  decision  maker’s  value  hierarchy  is  provided  in  Figure  70. 
There  are  only  two  evaluation  measures,  cost  and  time.  Both  are  uncertain.  Two 
alternatives  have  been  identified.  Alternative^  is  to  employ  in-house  construction  assets. 
Alternative  B  is  to  hire  a  contractor.  Alternative  A  is  less  expensive,  but  will  take 
considerably  longer.  Evaluation  measure  distributions  are  as  provided  by  Law  and 
Kelton  (1991,  331  -  334).  The  cost  of  Alternative^,  measured  in  $k,  is  estimated  to 
behave  in  accordance  with  the  gamma(3,2)  distribution.  The  cost  of  Alternative  B  is 
estimated  by  the  Weibull(5,10).  Time  for  project  completion,  measured  in  weeks,  is 
estimated  to  follow  the  Weibull(3,8)  distribution  for  Alternative  A  and  the  gamma  (2,1) 
for  Alternative  B.  The  relative  weights  of  cost  and  time  have  been  determined  to  be 
0.667  and  0.333,  respectively.  The  ranges  for  cost  and  time  are  [0,  20]  for  both 
evaluation  measures.  These  data  and  the  preference  functions  are  summarized  in  Table 
19.  As  per  the  class  of  problems  under  discussion,  the  utility  functions  are  unknown, 
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Cost 


Time 


Alternative  A 


Alternative  B 


Figure  70.  Value  Hierarchy  and  Evaluation  Measures  for  Parking  Lot  Example. 


however  they  are  listed  in  Table  19  for  this  example.  They  are  exponential  in  form,  with 
px  =  8  and  /?,  =  2  .  The  balance  of  the  information  is  known.  The  issue  is  which,  if  any, 

of  the  utility  functions  must  be  discerned  in  order  to  adequately  model  the  problem  within 
the  desired  accuracy.  The  value/utility  estimate  should  agree  to  within  0. 10  of  the  true 
problem  utility. 

While  this  problem  is  small,  with  only  two  evaluation  measures,  the  class  of 
problems  of  interest  consists  of  problems  with  many  evaluation  measures.  Often  the 
influential  evaluation  measures  are  a  small  subset.  This  is  recognized  in  decision  analysis 
literature  as  well  as  response  surface  methodology  literature.  In  RSM,  the  Scarcity-of- 
Effect  Principle  indicates  that  a  system  is  usually  significantly  influenced  by  a  few  main 
effects  and  low-order  interactions  (Myers  and  Montgomery,  1995:  134).  For  efficiency 
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Table  19.  Parking  Lot  Example  Information. 


Cost,  x. 

Time,  x2 

Alternative  A,  in-house 

gamma(3,2) 

Weibull(3,8) 

Alternative  B,  contractor 

Weibull(5,10) 

gamma(2,l) 

Range 

ro,  201 

ro,  201 

Value  Function 

v,U)  =  l-)4 

-(20-*)/ 

,  ,  1  —  e  Ao 

m*2)=  _20/ 

1  —  e  Ao 

Utility  Function 
(unknown) 

-(20-*)/ 

,  \  1-e  A 

Ul  (V)=  -20/ 

1-e  /s 

-(20— x2) / 

..  A 
U2\X2)  -20/ 

\-e  /2 

Weight 

wl  =  0.667 

w2  =  0.333 

in  analysis  it  is  desirable  to  screen  the  variables  to  reduce  those  under  consideration  by 
elimination  of  those  that  fail  to  influence  the  problem. 

In  RSM,  screening  experiments  typically  fit  a  first  order  polynomial  that 
approximates  the  response  to  the  predictor  variables.  Screening  experiments  reduce  the 
number  of  variables  by  identifying  those  that  do  not  significantly  affect  the  system  under 
consideration.  These  nonsignificant  variables  may  be  eliminated  from  consideration. 
Typically  in  RSM,  additional  analysis  is  conducted  employing  higher  order  polynomials 
to  better  fit  the  response  surface.  Such  modeling  is  not  required  for  the  hybrid  value- 
utility  algorithm. 

For  this  problem  a  pre-screen  was  used  where  some  stochastic  variables  were 
treated  in  a  deterministic  fashion  as  an  excursion.  Normally  variables  are  not  treated 
deterministically.  Additionally,  the  small  number  of  variables  in  this  example  problem 
does  not  require  group  screening  techniques,  so  they  are  omitted. 

Pre-Screen  Phase.  Prior  to  screening  the  variables,  a  pre-screening  experiment 
was  performed  as  an  excursion.  Rather  than  examining  the  stochastic  variables,  they 
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were  treated  deterministically  as  an  approximation.  The  intent  was  to  compare  the  results 
of  this  deterministic  approximation  to  a  nonnal  screening  approach. 

Because  this  example  problem  has  few  variables,  no  group  screening  is  required. 
Typical  experimental  designs  are  sufficient.  The  pre-screen  design  selected  was  a  2k 
plus  center  point.  This  design  consists  of  each  variable  set  at  two  levels,  high  and  low, 
plus  the  center  point,  the  mid  point  between  all  variables.  The  design  is  referred  to  as  a 
2k  design  as  the  number  of  required  runs  is  equal  to  2k  where  k  is  the  number  of 
variables,  plus  the  center  point  runs.  The  cost  and  time  variables  were  treated  in  a 
detenninistic  fashion  for  the  prescreen  analysis.  The  high  and  low  values  were  those 
associated  with  probabilities  of  0.4  and  0.6,  respectively.  That  is,  each  xj  were  selected 

such  that  |  '  /  (y)dy  =  0.4  for  the  low  point  (represented  by  and  J  '  /  (y)dy  =  0.6 

for  the  high  (represented  by  “+”)  point.  The  midpoint  (represented  by  “0”)  was  xi  such 

that  |  '  / (y  )dy  =  0.5  .  These  probabilities  produce  different  xi  for  the  two  alternatives, 

so  an  indicator  variable,  a,  was  used  to  designate  which  alternative  pertained  to  each  run. 
The  utility  functions  were  varied  by  changing  the  coefficient  of  the  exponential  function, 
p ,  for  each  evaluation  measure.  For  the  high  point,  /?  =0.1.  For  the  low  point, 

p '.  =  -0.1  At  the  midpoint,  pi=°°.  The  input  data  is  displayed  in  Table  20.  The 

symbols  “+”  indicate  that  the  high  level  of  the  variable  is  to  be  used,  the  low  level, 
and  “0”  indicates  the  midpoint  or  nominal  datum. 

For  each  row  of  the  experimental  design  matrix,  the  variables  are  set  at  either  the 
high  or  low  level,  or  at  the  center  point.  The  variables  are  inserted  in  the  utility  function 
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Table  20.  Pre-Screen  Experiment  Input  Data. 


Alternative,  a 

A 

B 

Cost,  xl 

0 

5.3905 

9.285 

+ 

6.211 

9.827 

- 

4.57 

8.743 

Time,  x2 

0 

7.0825 

1.67 

+ 

7.77 

2.022 

- 

6.395 

1.376 

Utility,  pvp2 

0 

OO 

OO 

+ 

0.1 

0.1 

- 

-0.1 

-0.1 

where  the  value  function  is  as  indicated  in  Table  19  and  the  utility  function  is 

-h-v.-o,'))/ 

¥,(eU))  =  -  g  -1/ -  (124) 

\-e  /pi 

and  the  objective  function  is 

U  =  Yjwi-ui(vi(xi)).  (125) 

1=1 

The  design  matrix  and  the  results  are  presented  in  Table  21.  The  variable  settings 
are  indicated  in  order  of  the  cost,  time,  cost  utility  exponential  coefficient,  time  utility 
exponential  coefficient,  and  alternative.  As  above,  the  variable  setting  of  high  is 
indicated  by  a  plus  sign,  low  by  a  minus  sign,  and  center  points  by  a  zero. 

A  least  squares  model  was  fitted  to  the  U  data  in  Table  21.  Figure  71  shows  the 

observed  data,  U  ,  plotted  against  the  predicted  values,  0  .  The  two  points  that  lie 
outside  of  the  95  percent  confidence  bounds  are  the  center  points,  and  indicate  lack  of  fit 
of  the  model  to  the  data.  Despite  this,  the  R 2  value  for  the  model  was  0.99,  the  adjusted 
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Table  2 1 .  Pre-Screening  Experimental  Design  and  Results. 


Row 

Pattern 

U 

Row 

Pattern 

u 

1 

0.068 

18 

+— + 

0.00488 

2 

-— + 

0.00840 

19 

+--+- 

0.337 

3 

— +- 

0.375 

20 

+ 

i 

i 

+ 

+ 

0.12 

4 

— ++ 

0.123 

21 

+-+-- 

0.667 

5 

-+- 

0.667 

22 

+-+-+ 

0.663 

6 

--+-+ 

0.665 

23 

+-++- 

0.974 

7 

--++- 

0.974 

24 

+-+++ 

0.778 

8 

--+++ 

0.78 

25 

++— 

0.03 

9 

-+— 

0.68 

26 

+ 

+ 

i 

i 

+ 

0.00488 

10 

-+--+ 

0.00841 

27 

++-+- 

0.351 

11 

0.389 

28 

++-++ 

0.163 

12 

-+-++ 

0.166 

29 

+++-- 

0.667 

13 

-++-- 

0.667 

30 

+++-+ 

0.663 

14 

-++-+ 

0.667 

31 

++++- 

0.988 

15 

-+++- 

0.988 

32 

+++++ 

0.821 

16 

-++++ 

0.823 

33 

0000+ 

0.414 

17 

+— - 

0.03 

34 

0000- 

0.625 

0. 

0. 

U  Observed 

0. 

0. 


.0  .1  .2  .3  .4  .5  .6  .7  .8  .9  1.0 

U  Predicted 

Figure  7 1 .  Prediction  Versus  Actual  Data  Plot  for  Pre-Screen  Experiment. 
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R2  was  0.97.  Despite  the  lack  of  fit  at  the  center  points,  analysis  was  continued.  The 
analysis  of  variance  (ANOVA)  results  are  provided  in  Table  22.  The  p  value  for  the  F 
test  indicates  that  the  model  is  highly  significant.  This  means  that  the  model  provides  a 
significantly  better  estimate  of  the  problem  than  does  the  overall  mean.  Parameter 
estimates  are  contained  in  Table  23.  The  two  utility  coefficients  were  significant, 
indicating  that  the  utility  functions  are  important,  at  least  at  the  margins  of  the  utility 
function  envelope.  Had  this  been  a  large  model,  likely  not  all  utility  functions  would 
have  been  significant,  permitting  removing  them  from  consideration  for  the  hybrid  utility 
model.  The  alternatives  were  significant.  Only  one  interaction,  the  time  utility  and  the 
alternative  were  significant. 


Table  22.  ANOVA  Results  from  Model  Fitted  to  Pre-Screen  Data. 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

15 

3.80 

0.253 

80.285 

Error 

18 

0.0569 

0.00316 

Prob  >  F 

C.  Total 

33 

3.86 

<.0001 

Screening  Experiment.  A  standard  screening  experiment  was  conducted.  As 
mentioned  in  the  prescreen  section,  the  few  variables  of  this  example  mean  that  no 
groups  screening  is  required.  Typical  experimental  designs  where  the  variables  are 
considered  individually  are  sufficient.  The  experimental  design  was  a  full  factorial  plus 
center  point  replications.  The  center  point  was  repeated  twice  with  each  alternative,  for  a 
total  of  four  center  point  runs.  The  cost  and  time  evaluation  measures  were  handled 
normally  as  random  variables.  The  variables  in  the  experiment  are  the  exponential 
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Table  23.  Parameter  Estimates.  Variable  x3  represents  cost,  x0  time,  x3  the  cost  utility 
coefficient,  x4  the  time  utility  coefficient,  x5  and  the  alternative,  a. 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>  t 

Intercept 

0.463 

0.00964 

48.02 

<.0001 

Cost 

-0.00541 

0.00994 

-0.54 

0.593 

Time 

0.00713 

0.00994 

0.72 

0.483 

Pcost 

0.319 

0.00994 

32.09 

<.0001 

P Time 

0.113 

0.00994 

11.33 

<.0001 

a 

0.0462 

0.00964 

4.80 

0.0001 

Cost  *  Time 

3.13e-8 

0.00994 

0.00 

1.0000 

Cost  *  Pcost 

0.00491 

0.00994 

0.49 

0.628 

Time  *  pCost 

-3.44e-7 

0.00994 

-0.00 

1.0000 

Cost  *  pTime 

0.0000328 

0.00994 

0.00 

0.997 

Time  *  PTime 

0.00712 

0.00994 

0.72 

0.483 

P Cost  *  Prime 

0.000018 

0.00994 

0.00 

0.999 

Cost  *  a 

-0.00409 

0.00994 

-0.41 

0.685 

Time  *  a 

-0.00363 

0.00994 

-0.36 

0.719 

Pcost*  a 

-0.00986 

0.00994 

-0.99 

0.334 

Prime*  0 

0.0444 

0.00994 

4.47 

0.0003 

coefficients  for  the  evaluation  measures,  the  cost  weight  variable,  and  a  variable  assigned 
for  the  alternative.  Only  a  single  weight  variable  may  be  employed,  as  the  weights  must 
sum  to  one.  Including  the  time  weight  causes  colinearity  problems.  The  high  and  low 
levels  for  the  cost  weight  were  ±10  percent  of  the  nominal  (elicited)  value,  and  the  time 
weight  adjusted  accordingly.  Table  24  displays  the  variable  values  for  the  screening 
experiment. 

The  experimental  design  is  illustrated  in  Table  25,  along  with  the  results  of  each 
run.  The  experiment  was  not  randomized,  as  is  typically  done  in  an  experiment  involving 
physical  systems,  as  the  results  are  completely  from  mathematical  expressions  and 
random  number  generation.  The  pattern  column  of  Table  25  indicates  the  settings  of  the 
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Table  24.  Screening  Experiment  Input  Data. 


Variable 

Position 

Value 

Utility,  px,p2 

Center  point 

OO 

+ 

0.1 

- 

-0.1 

Cost  Weight,  wx 

Center  point 

0.667 

+ 

0.7337 

- 

0.6003 

Alternative 

+ 

B 

- 

A 

Table  25.  Screening  Experimental  Design  and  Results.  Row  indicates  experimental 
design  points.  Pattern  indicates  the  condition  of  the  variables  px ,  p2,  the  alternative,  and 

wx .  The  observed  utility  is  U  ,  the  predicted  utility  U  . 


Row 

Pattern 

U 

U 

Row 

Pattern 

u 

U 

1 

— 

0.0000356 

0.0298 

11 

+-+- 

0.595 

0.690 

2 

— + 

0.000693 

0.0885 

12 

+-++ 

0.731 

0.768 

3 

-+- 

0.00720 

0.0444 

13 

++- 

0.938 

0.962 

4 

-++ 

0.00613 

0.0908 

14 

++-+ 

0.858 

0.952 

5 

-+- 

0.229 

0.323 

15 

+++- 

0.727 

0.770 

6 

-+-+ 

0.268 

0.282 

16 

++++ 

0.67 

0.749 

7 

-++- 

0.048 

0.126 

17 

00+0 

0.71 

0.508 

8 

-+++ 

0.019 

0.0723 

18 

00-0 

0.677 

0.414 

9 

+ — 

0.591 

0.669 

19 

00+0 

0.758 

0.508 

10 

+--+ 

0.729 

0.759 

20 

00-0 

0.657 

0.414 

four  variables  in  the  order  of  pCost ,  pTime ,  alternative,  and  weight.  A  minus  sign  indicates 

that  the  variable  for  that  position  is  at  the  low  setting.  The  high  setting  is  indicated  by  a 
plus  sign.  A  zero  indicates  that  the  variable  is  to  be  placed  at  the  center  point  setting. 
Table  25  also  contains  the  results  of  each  experimental  run,  the  observed  utility  U  .  For 
ease  of  comparison,  the  utility  estimated  by  the  fitted  least  squares  model,  U  is  also 
contained  in  Table  25. 
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The  predicted  versus  observed  utility  data  is  shown  in  Figure  72.  The  R2  and 
adjusted  R2  values  for  the  model  were  0.863  and  0.712,  respectively.  The  four  points 
outside  of  the  95  percent  confidence  bounds  are  the  four  center  points.  The  two  center 
points  for  alternative  A  are  the  vertical  pair  on  the  right.  The  other  pair  is  from  center 
point  runs  with  alternative  B.  ANOVA  results  are  presented  in  Table  26.  The  model  is 
highly  significant  when  compared  to  the  overall  mean  response. 


Figure  72.  Prediction  Versus  Actual  Data  Plot  for  Screening  Experiment. 


Table  26.  ANOVA  Results  from  Model  Fitted  to  Screening  Data. 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Model 

10 

1.9071229 

0.190712 

5.6922 

Error 

9 

0.3015372 

0.033504 

Prob  >  F 

C.  Total 

19 

2.2086602 

0.0076 
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Table  27  contains  the  lack  of  fit  statistics.  The  lack  of  fit  is  significant,  indicating 
that  the  model  likely  could  be  improved  through  the  addition  of  higher  order  terms  of  the 
model  predictor  variables.  This  is  not  unusual  for  screening  experiments. 


Table  27.  Lack  of  Fit  Statistics. 


Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Ratio 

Lack  Of  Fit 

7 

0.30018524 

0.042884 

63.4373 

Pure  Error 

2 

0.00135200 

0.000676 

Prob  >  F 

Total  Error 

9 

0.30153724 

0.0156 

Max  RSq 

0.9994 

The  estimates  of  the  parameters  are  provided  in  Table  28.  The  exponential 
constant  for  the  cost  utility  function  is  highly  significant.  This  indicates  that  the  true  cost 
utility  function  has  the  potential  to  be  significant  in  the  hybrid  model.  In  contrast,  the 
time  utility  function  is  not  significant.  This  indicates  that  within  the  range  of  exponential 
constant  values  normally  encountered,  the  time  utility  function  may  be  represented  by  the 
time  value  function. 

The  parking  lot  example  was  analyzed  using  Logical  Decisions®  software.  In 
succession  the  single  dimensional  preference  functions  used  in  the  model  were  both  value 
functions.  Then  the  significant  cost  function  was  replaced  with  the  appropriate  single 
dimension  utility  function.  This  completes  the  hybrid  model.  For  comparative  purposes, 
the  other  two  possible  model  combinations  were  examined.  These  are  the  full  utility 
model,  with  utility  functions  used  for  both  single  dimensional  preference  functions,  and 
the  remaining  possible  hybrid  model  combination,  with  cost  modeled  with  a  value 
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Table  28.  Screening  Experiment  Parameter  Estimates. 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>  t 

Intercept 

0.461 

0.0409 

11.26 

<.0001 

P  Cost 

0.329 

0.0458 

7.19 

<.0001 

P  Time 

0.0686 

0.0458 

1.50 

0.168 

Alternative 

0.0472 

0.0409 

1.15 

0.278 

Weight 

0.00916 

0.0458 

0.20 

0.846 

Pcost  *  Prime 

-0.000184 

0.0458 

-0.00 

0.997 

Pcost  *  Alternative 

-0.00153 

0.0458 

-0.03 

0.974 

Prime  *  Alternative 

0.0530 

0.0458 

1.16 

0.277 

Post  *  Weight 

0.00796 

0.0458 

0.17 

0.866 

Prune  *  Weight 

-0.0250 

0.0458 

-0.55 

0.598 

Alternative  *  W eight 

0.00305 

0.0458 

0.07 

0.948 

function  and  tine  modeled  with  an  utility  function.  These  final  two  models  are  not  called 
for  in  the  algorithm  and  so  would  not  be  considered  when  employing  the  methodology 
for  a  complex  decision.  These  results  are  presented  in  Table  29.  In  each  case  Alternative 
A  is  the  best  choice. 


Table  29.  Results  for  Various  Combinations  of  Value  and  Utility  Functions  for  the 

Parking  Lot  Example. 


Sequence 

Preference 

Functions 

Alternative  A 

Alternative  B 

Model 

Results 

Difference 
from  Utility 
Model 

Model 

Results 

Difference 
from  Utility 
Model 

1 

Value:  Cost 
and  Time 

0.742 

0.177 

0.682 

0.183 

2 

Utility:  Cost 
Value:  Time 

0.862 

0.057 

0.853 

0.012 

Not 

Applicable 

Utility:  Cost 
and  Time 

0.919 

0 

0.865 

0 

Not 

Applicable 

Utility:  Time 
Value:  Cost 

0.799 

0.12 

0.694 

0.171 
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Examination  the  sensitivity  of  the  alternatives  to  the  relative  importance  (weight) 
of  the  evaluation  measures  is  shown  in  Figure  73  through  Figure  76.  Figure  73  depicts 
the  sensitivity  of  the  exclusively  value  model.  Figure  74  illustrates  the  sensitivity  of  the 
algorithm’s  hybrid  model.  Figure  75  shows  the  sensitivity  of  the  model  using  only  utility 
functions.  The  remaining  possible  combination,  which  is  not  called  for  in  the  algorithm, 
is  shown  in  Figure  76.  Figure  76  most  closely  resembles  the  full  utility  model.  The 
vertical  line  in  each  graph  indicates  the  nominal  cost  weight  value,  wCost  =  0.667  .  The 

RSM  examination  only  varied  wCost  by  ten  percent.  The  alternative  choice  is  not 
sensitive  within  this  range  for  the  value  model  (Figure  73)  and  the  utility  model  (Figure 
75).  However  the  hybrid  model  (Figure  74)  is  sensitive  to  the  weights.  While  the  hybrid 
model  incorporates  the  significant  utility  aspects,  the  response  to  other  perturbations  is 
not  representative  of  the  utility  model  (or  the  value  model  either).  This  is  a  shortcoming 
in  the  methodology,  but  it  can  be  mitigated  through  performance  of  sensitivity  analysis 
on  both  the  hybrid  model  and  the  value  model. 


-  A,  In  House 

-  B,  Contractor 


Figure  73.  Weight  Sensitivity  for  Parking  Fot  Example  with  Single  Dimensional  Value 

Functions. 
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-  A,  In  House 

-  B,  Contractor 


Figure  74.  Weight  Sensitivity  for  Parking  Lot  Example  with  Single  Dimensional  Cost 

Utility  and  Time  Value  Functions. 


-  A,  In  House 

-  B,  Contractor 


Figure  75.  Weight  Sensitivity  for  Parking  Lot  Example  with  Single  Dimensional  Utility 

Functions. 


-  A,  In  House 

-  B,  Contractor 


Figure  76.  Weight  Sensitivity  for  Parking  Lot  Example  with  Single  Dimensional  Cost 

Value  and  Time  Utility  Functions. 
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Example  Summary.  The  parking  lot  example  problem  has  been  investigated 
using  both  a  prescreen  approach,  where  the  variables  were  treated  deterministically,  and 
with  a  stochastic  screening  methodology.  The  two  approaches  are  depicted  graphically  in 
Figure  77.  The  prescreen  indicated  that  both  the  utility  functions  may  be  significant 
aswell  as  the  alternative  and  one  pair  of  interactions.  The  screening  experiment  identified 
only  the  cost  utility  function  as  being  potentially  significant.  The  screening  experiment 
was  the  more  accurate  approach.  The  optimum  alternative  never  did  change,  as  predicted 
by  the  screening  experiment  in  contrast  with  the  prescreen.  The  hybrid  approach  did 
perform  potentially  poorly  with  respect  to  the  sensitivity  of  the  hybrid  model  compared  to 
the  true  utility  models.  However  the  performance  from  an  expected  value  perspective 
was  satisfactory. 


Prescreen  Screening 

Experiment  Experiment 


Figure  77.  Prescreen  and  Screening  for  Parking  Lot  Example. 
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Demonstration  of  Value-Utility  Hybridization 


Klimack,  Bassham,  and  Bauer  (2002)  provide  an  example  of  application  of 
decision  analysis  to  an  Automatic  Target  Recognition  (ATR)  problem.  This  application 
of  decision  analysis  included  eliciting  both  single  dimensional  value  and  utility  functions 
from  the  subject,  an  expert  in  the  area.  This  provides  an  opportunity  to  employ  the 
Value-Utility  Hybridization  methodology.  The  basic  DA  analysis  is  provided  in 
Appendix  D.  A  brief  summary  is  provided  for  background  here. 

Background.  Automatic  Target  Recognition  (ATR)  is  a  processing  problem 
where  an  image  is  examined  to  identify  or  otherwise  characterize  targets.  All  ATR 
classification  systems  (CSs)  have  a  number  of  desirable  traits  that  may  be  used  as 
evaluation  measures  when  selecting  from  among  candidate  systems.  In  general,  these 
measures  have  not  applied  in  total,  but  specific  measures  have  been  selected  when 
considering  alternatives  for  a  specific  program.  A  decision  analysis  approach  was 
employed.  A  value  model  was  constructed.  The  value  hierarchy  is  presented  in  Figure 
78,  and  captures  the  motivators  important  to  the  decision  maker.  Besides  the  values,  the 
problem  is  affected  by  the  two  mission  profiles  (combat  identification  [CID]  and 
intelligence/surveillance/reconnaissance  [ISR])  under  which  the  ATR  CS  would  be 
employed.  Additionally  there  are  three  alternatives  under  consideration.  Employing 
single  dimensional  value  functions  and  single  dimensional  utility  functions  produced 
differing  results,  indicating  that  the  problem  was  sensitive  to  preference  function 
construction. 
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ATR 


Figure  78.  The  Value  Hierarchy  for  the  ATR  System.  The  dashed  line  under  Overall 
Detection  Perfonnance  indicates  that  either  the  Defined  Pd  or  the  ROC  measures  are 

employed.  The  parenthetical  numbers  indicate  the  relative  weights  for  a  value,  within  the 

parent  value. 


Employing  RSM  to  Prioritize  Model  Parameters.  The  ATR  DA  problem  has 
23  evaluation  measures,  23  corresponding  weights,  two  mission  profiles  (combat 
identification  [CID]  and  intelligence/surveillance/reconnaissance  [ISR]),  and  three 
alternatives.  This  is  a  moderately  large  number  of  parameters.  As  the  weights  sum  to 
unity,  they  may  not  simply  be  perturbed  in  the  RSM  process.  The  procedure  provided  by 
Kirkwood  (1997b:  83  -  84)  and  implemented  by  Bauer,  Parnell,  and  Meyers  (1999:  168) 
was  employed.  The  relative  weight  deemed  most  important,  wt ,  (generally  the  largest  in 

magnitude)  was  varied  by  a  factor  of  0. 1 ,  or  h;+  =  1 . 1  wj  and  w~  =  0.9 wi .  The  remaining 

weights  were  adjusted  to  meet  the  summation  constraint  while  retaining  their  relative 
ratios.  The  weights  then  become  a  single  variable  in  the  regression  to  find  the  response 
surface. 

The  remaining  25  variables  (the  23  evaluation  measure  variables,  the  mission 
profile,  and  the  alternative  variables)  are  still  a  large  number  of  parameters.  To  reduce 
the  number  to  a  more  tractable  amount,  the  group  screening  technique  was  employed. 
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Bauer,  Parnell,  and  Meyers  (1999:  165  -  166)  introduced  this  approach  for  the  decision 
analysis  context.  In  this  method,  the  input  variables  are  grouped  in  some  fashion. 
Perturbations  are  made  uniformly  within  each  group,  that  is,  all  group  members  are  set  at 
their  high  or  low  setting  simultaneously.  If  a  variable  group  is  found  not  to  be 
significant,  then  all  group  members  are  not  considered  significant.  The  group  screening 
results  are  independent  of  how  the  grouping  is  perfonned,  but  in  a  DA  context  the  value 
hierarchy  offers  an  obvious  cladistical  grouping  tool. 

In  this  application,  the  number  of  evaluation  measures  suggests  that  group 
screening  will  be  employed  iteratively.  Here  the  first  decomposition  of  the  value 
hierarchy,  the  second  row  from  the  top,  will  be  used  as  an  initial  groups  screening.  RSM 
will  be  employed  on  these  groups  to  detennine  their  significance.  RSM  will  then  be 
employed  separately  for  each  decomposition  of  the  value  groups  from  the  second  row. 
The  evaluation  measure  least  square  coefficients  from  the  RSM  analyses  will  then  be 
combined  to  produce  a  single  list  of  significant  evaluation  measures.  The  list  will  be 
prioritized  based  on  least  squares  model  coefficients.  This  prioritized  list  will  then  be 
used  for  construction  of  the  hybrid  value-utility  model,  which  is  then  used  to  analyze  the 
decision  in  a  typical  manner. 

Initial  Group  Screening.  For  the  ATR  analysis,  the  grouping  of  the  continuous 
variables  was  selected  at  the  second  level  of  the  value  hierarchy  (see  Figure  79).  This 
provides  seven  groups: 

1 .  Robustness, 

2.  Overall  Detection  Perfonnance, 
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3.  Employment  Concept, 

4.  Declaration  Ability, 

5.  Classification  Ability, 

6.  Cost,  and 

7.  Self-Assessed  Accuracy. 

Counting  the  weight  variable,  there  were  eight  continuous  variables.  There  are  also  two 
categorical  variables:  the  mission  profile,  with  two  levels,  CID  and  ISR;  and  the 
alternative,  with  three  levels,  ATR  33,  ATR  55,  and  ATR  89.  The  design  was  developed 
in  the  JMP®  software  package,  which  recommended  a  L36  Taguchi  Design.  The 
categorical  variables  reduce  the  flexibility  in  choosing  a  design.  For  this  example  only 
first  order  effects,  without  interactions,  will  be  examined.  As  no  interactions  are  under 
consideration,  aliasing  concerns  are  avoided. 


ATR 


Robustness 

Overall 

Employment 

Declaration 

Classification 

Cost 

Self  Assessed 

(0.2) 

Detetction 

Concept 

Ability 

Ability 

(0.10) 

Accuracy 

Performance 

[Rating] 

[Pdec] 

(0.11) 

(0.14) 

(0.17) 

(0.15) 

(0.13) 

Figure  79.  First  Group  Screening,  Second  Tier  of  Value  Hierarchy. 


An  alternative  approach  to  using  an  experimental  design  that  includes  categorical 
variables  would  be  to  select  a  specific  combination  of  categories.  For  this  problem, 
perhaps  the  CID  profile  and  the  ATR  55  alternative.  The  problem  is  then  reduced  to 
continuous  input  variables,  which  lend  themselves  to  traditional  fractional  factorial 
designs.  Working  strictly  with  continuous  variables  also  permits  examination  of 
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interactions,  which  was  not  considered  here,  because  of  the  increased  availability  of 
orthogonal  experimental  designs.  Six  center  points  were  added  to  the  L36  design, 
pennitting  the  various  combinations  of  profile  and  alternative  to  be  examined. 

The  L36  design  employs  the  continuous  variables  at  a  high  or  low  setting, 
indicated  by  a  plus  or  minus  sign,  respectively,  in  the  pattern  notation.  The  patterns  for 
the  matrix  rows  are  shown  in  Table  30.  For  the  continuous  variables,  the  minus  sign 
indicates  the  variable  of  interest  is  set  to  the  lower  level,  a  plus  sign  indicates  it  is  set  to 
the  higher  level.  The  number  zero  identifies  center  points.  For  the  categorical  variables, 
which  are  handled  by  JMP  with  dummy  variables,  the  minus,  plus,  and  zero  indicate 
different  categories.  For  example,  the  lead  minus  sign  in  row  1  indicates  that  the 
robustness  variable  is  at  the  lower  setting.  The  order  of  the  variables  is  as  presented  in 
the  list  above,  then  the  profile  and  alternative  variables.  The  data  for  these  setting  are 
presented  below  in  Equations  (127)  through  (131).  More  detailed  infonnation  is 
available  in  Appendix  E.  Also  shown  in  Table  30  are  the  results  for  each  combination 

(row),  U . 

To  determine  the  response,  the  decision  problem  was  modeled  in  the  DPL® 
software  package.  The  various  evaluation  measure  levels,  xt ,  were  passed  to  an  Excel 

spreadsheet  where  v(.  (x, )  was  calculated.  This  intermediate  result  was  used  to  determine 

«,(';,(a))  from 

~Vi(x/ 

»,Ma))=  1  g  -i/  (126) 

\-e  'Pi 
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Table  30.  Initial  Group  Screening  Design  and  Response  for  ATR  Problem. 


Row 

Pattern 

0 

Row 

Pattern 

0 

Row 

Pattern 

u 

1 

0.278 

15 

+-++.+ — + 

0.627 

29 

— +++.++0 

0.360 

2 

- o 

0.302 

16 

++-++.+ — 

0.610 

30 

— +++-+++ 

0.376 

3 

- + 

0.292 

17 

++-++-+— 0 

0.623 

31 

+ — +++-+- 

0.524 

4 

+.+ — +++. 

0.535 

18 

++-++-+--+ 

0.602 

32 

+— +++-+0 

0.515 

5 

+-+— +++0 

0.505 

19 

+++-++-+- 

0.772 

33 

+—+++-++ 

0.531 

6 

+.+ — ++++ 

0.506 

20 

+++-++-+-0 

0.806 

34 

-+ — +++- 

0.414 

7 

++.+ — ++. 

0.480 

21 

+++-++-+-+ 

0.795 

35 

-+— +++-0 

0.433 

8 

++-+ — ++0 

0.544 

22 

-+++-++-+- 

0.512 

36 

-+ — +++-+ 

0.414 

9 

++-+ — +++ 

0.546 

23 

-+++-++-+0 

0.565 

37 

00000000- 

0.509 

10 

-++-+ — +- 

0.601 

24 

-+++-++-++ 

0.583 

38 

00000000-0 

0.556 

11 

-++-+— +0 

0.645 

25 

_+++-++- 

0.500 

39 

00000000-+ 

0.525 

12 

-++-+ — ++ 

0.647 

26 

— +++-++-0 

0.545 

40 

00000000+- 

0.497 

13 

+-++-+ — 

0.574 

27 

--+++-++-+ 

0.541 

41 

00000000+0 

0.531 

14 

+-++-+ — 0 

0.622 

28 

-+++-++- 

0.381 

42 

00000000++ 

0.497 

where 


f  0.1 


Pi  =  \ 


oo 


[-0.1 


for  pattern  "+" 
for  pattern  "0" 
for  pattern 


A  zero  indicates  nominal  setting,  when  ut  (v,  (x,- ))  =  vi  (x,. ) . 


(127) 


The  weight  variable  w  is  a  vector.  The  nominal  value  for  the  largest  weight  is 
that  of  the  Robustness  evaluation  measure  with  a  weight  of  0.2.  However  no  data  was 
available  for  the  robustness  for  identification  of  class.  Because  of  this,  the  next  largest 
evaluation  measure  weight  was  selected.  This  was  the  Overall  Detection  Performance 
evaluation  measure,  denoted  w2,  equal  to  0.17.  Perturbing  this  by  0.1,  or 
w2  =  w2±  0. 1  vv2 ,  and  adjusting  the  remaining  weights  produces  two  weight  vectors  as 
called  for  by  the  high  and  low  pattern  indicators.  As  with  pi ,  a  zero  indicates  nominal 
setting,  or  the  elicited  weights,  w° . 
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These  vectors  are 


where 


w  =  < 


w 

w° 

w“ 


for  pattern  "+" 
for  pattern  "0" 
for  pattern 


"0.1959” 

'0.2' 

'0.2041' 

0.187 

0.17 

0.153 

0.1469 

0.15 

0.1531 

0.1273 

,  w°  = 

0.13 

,  w  = 

0.1327 

0.1077 

0.11 

0.1123 

0.098 

0.1 

0.1020 

0.1371 

0.14 

0.1429 

(128) 


(129) 


The  categorical  variables,  which  were  handled  by  employing  dummy  variables,  were  set 
according  to 


Profile  = 


JlSR 

[CID 


for  pattern  "+" 
for  pattern 


(130) 


and 


Alternative  =  < 


ATR  89 
ATR55 
ATR  33 


for  pattern  "+" 
for  pattern  "0" 
for  pattern 


(131) 


The  response  variable,  utility  U  ,  is  also  presented  in  Table  30.  A  plot  of  actual 
versus  predicted  values  is  contained  in  Figure  80.  The  linear  model  fit  well  with 
R2  =  0.978  and  R2dj  =  0.970  .  The p  value  was  less  than  0.0001.  The  parameters 

estimates  and  the  significance  information  for  the  model  are  contained  in  Table  3 1 .  The 
Declaration  Ability  and  Self-Assessment  groups  are  clearly  not  significant.  They  may  be 
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dropped  from  consideration.  The  weights  are  also  not  significant,  which  agrees  with  the 
sensitivity  analysis  performed  as  part  of  the  basic  DA  analysis  of  the  problem,  and  will  be 
eliminated  from  further  consideration.  The  other  groups  are  clearly  significant.  Note  that 
the  coefficient  estimates  for  the  least  squares  model  agree  with  the  significance 
information.  The  significant  variables  have  coefficients  that  are  an  order  of  magnitude 
larger  than  those  found  to  be  non-significant.  The  coefficients  also  provide  a  measure  of 
importance.  Repeating  the  model  fit  with  these  groups  removed  would  provide  the  same 
information  (although  the  p-values  would  change  slightly  because  of  changes  in  the 
number  of  degrees  of  freedom)  and  so  is  not  required. 


Figure  80.  ATR  Group  Screening  Model  Actual  versus  Predicted  Values. 


The  significance  information  in  Table  3 1  may  be  used  to  prioritize  the  order  of 
converting  the  single  dimensional  value  functions  to  their  utility  equivalents.  However 
the  group  screening  has  only  eliminated  five  evaluation  measures.  Further,  four  groups 
have  their  significance  levels  indicated  as  equal.  This  fails  to  prioritize  between  eight 


194 


Table  3 1 .  ATR  Group  Screening  Parameter  Estimates. 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob  >  |t| 

Intercept 

0.529 

0.00316 

167.50 

<.0001 

Robustness 

0.0647 

0.00341 

18.95 

<.0001 

Detection  Perfonnance 

0.0577 

0.00341 

16.92 

<.0001 

Employment  Concept 

0.0738 

0.00341 

21.62 

<.0001 

Declaration  Ability 

0.00211 

0.00341 

0.62 

0.541 

Classification  Ability 

0.0456 

0.00341 

13.37 

<.0001 

Cost 

0.0139 

0.00341 

4.09 

0.0003 

Self  Assessment  Accuracy 

-0.00528 

0.00341 

-1.55 

0.132 

Weights 

-0.00556 

0.00341 

-1.63 

0.114 

CID  Profile 

0.0109 

0.00316 

3.46 

0.0016 

ISR  Profile 

-0.0109 

0.00316 

-3.46 

0.0016 

ATR  33  Alternative 

-0.0157 

0.00447 

-3.52 

0.0014 

ATR  55  Alternative 

0.0104 

0.00447 

2.32 

0.0274 

ATR  89  Alternative 

0.00536 

0.00446 

1.20 

0.240 

evaluation  measures.  For  further  refinement  the  next  step  is  to  decompose  the  groups,  to 
detennine  the  significance  of  the  group  members. 

Second  Group  Screening.  The  candidate  groups  for  decomposition  and 
examination  with  RSM  are:  Robustness,  Detection  Performance,  Classification  Ability, 
and  Cost.  The  Employment  Concept  criterion  is  not  decomposable,  so  cannot  be 
examined  further.  Employment  Concept  should  be  converted  to  utility  in  the  hybrid 
model.  (However  of  note  is  that  all  three  candidates  have  equal  scores  for  this  criterion. 
If  no  additional  candidates  are  introduced  with  a  differing  score  for  Employment 
Concept,  and  no  sensitivity  analysis  is  performed  on  the  Employment  Concept  scores,  it 
may  be  retained  as  a  value  function.  Typically  the  sensitivity  analysis  restriction  is 
onerous,  but  this  point  might  be  valuable  where  many  utility  elicitations  are  involved.) 
Eliminated  from  the  immediate  requirement  to  convert  from  value  to  utility  functions  are 
Declaration  Ability  and  Self-Assessment.  Choosing  the  first  subgroup  to  screen  should 
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be  done  with  consideration  as  to  the  potential  for  eliminating  criteria.  The  Cost  measure 
provides  this,  but  is  complex  with  two  sub-levels  and  eleven  evaluation  measures.  For 
simplicity  Robustness  is  chosen;  other  groups  will  be  addressed  later.  The  Robustness 
portion  of  the  value  hierarchy  is  shown  in  Figure  81. 


Figure  81.  Second  Group  Screening,  Robustness  Portion  of  Value  Hierarchy. 


Robustness  has  three  evaluation  measures:  Robustness  in  Detection  ( AP, ), 
Robustness  in  Typification  ( A Pid  ),  and  Robustness  in  Classification  ( A Pcc ).  The 
variables  under  consideration  are  the  exponential  constants  for  these  measures.  Refer  to 
them  as  px,p2,  and  p, ,  respectively,  for  this  portion  of  the  analysis.  They  take  on  values 

as  indicated  in  equation  (127).  The  weights  will  be  perturbed  as  above  with  the  largest 
weight,  that  of  Robustness  in  Detection,  being  perturbed  ten  percent  about  the  nominal 
value.  The  equivalent  of  equation  (129)  becomes 


"0.4675” 

”0.425” 

”0.3825” 

0.2547 

,  w°  = 

0.275 

,  w  = 

0.2953 

0.2778 

^  0.3 

0.3222 

(132) 
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The  profile  and  alternative  information  remains  unchanged.  Categorical  data 
again  restricts  the  possible  experimental  designs  available.  A  Hunter  Li 8  (SAS  Institute, 


Inc.,  2001:  57)  design  was  selected.  This  guarantees  orthogonality  and  there  are  no 
aliasing  concerns,  as  we  do  not  investigate  interactions  here.  The  Robustness  screening 
experimental  design  is  illustrated  in  Table  32,  along  with  the  results.  The  pattern 
information  indicates  variables  levels  as  explained  above,  with  a  variable  order  of 
Robustness  in  Detection,  Robustness  in  Typification,  and  Robustness  in  Classification, 
weight,  profile,  and  alternative.  Analysis  of  these  data  provides  the  coefficient  estimates 
in  Table  33. 


Table  32.  Robustness  Screening  Experimental  Design  and  Response  for  ATR  Problem. 


Row 

Pattern 

u 

Row 

Pattern 

u 

Row 

Pattern 

U 

1 

— ++- 

0.0289 

9 

-+-+0 

0.331 

17 

- + 

0.324 

2 

+++— 

0.949 

10 

++-+-0 

0.995 

18 

++++++ 

0.987 

3 

-++— 

0.577 

11 

-+-+-0 

0.534 

19 

000011 

0.618 

4 

+ 

i 

i 

+ 

+ 

l 

0.745 

12 

+-+-+0 

0.708 

20 

000012 

0.671 

5 

0.577 

13 

-+++++ 

0.526 

21 

000013 

0.584 

6 

+-++- 

0.745 

14 

+-— + 

0.701 

22 

000021 

0.618 

7 

-++-0 

0.286 

15 

— + 

0.324 

23 

000022 

0.671 

8 

++-+0 

0.996 

16 

++++++ 

0.987 

24 

000023 

0.584 

As  can  be  seen  from  the  data  in  Table  33,  Robustness  in  Detection  and 
Robustness  in  Typification  are  significant  while  Robustness  in  Classification  is  not 
significant.  The  first  two  evaluation  measures  are  candidates  for  conversion  from 
Vj  (.w )  to  Uj  (xy. )  while  the  last  is  excluded  from  consideration.  The  mission  profile  is 
not  significant.  Any  other  result  would  be  surprising,  as  the  input  data  for  robustness  was 
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Table  33.  ATR  Robustness  Screening  Parameter  Estimates. 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob  >  |t| 

Intercept 

0.628 

0.0127 

49.38 

<.0001 

Robustness  in  Detection 

0.226 

0.0149 

15.10 

<.0001 

Robustness  in  Typification 

0.140 

0.0149 

9.39 

<.0001 

Robustness  in  Classification 

0.0113 

0.0149 

0.75 

0.461 

Weights)- 1,1) 

-0.0214 

0.0149 

-1.43 

0.171 

Profile[CID] 

0.0109 

0.0129 

0.84 

0.411 

Profile[ISR] 

-0.0109 

0.0129 

-0.84 

0.411 

Alternative  [ATR  33] 

-0.0207 

0.0180 

-1.15 

0.267 

Alternative  [ATR  55] 

0.0213 

0.0180 

1.18 

0.254 

Alternative  [ATR  89] 

-0.000597 

0.0180 

-0.03 

0.974 

not  conditioned  on  mission  profile.  The  alternatives  are  not  significant.  This  indicates 
that  the  most  desired  alternative  changes  over  the  range  of  the  perturbation  introduced  in 
this  analysis. 

Third  Group  Screening.  Turning  to  consider  cost,  there  are  eleven  evaluation 
measures.  The  screening  design  must  include  twelve  continuous  variables  to  account  for 
the  evaluation  measures  and  the  weights  (see  Figure  82),  plus  a  three-level  categorical 
variable  for  the  alternatives.  The  profile  may  be  neglected,  as  costs  are  not  conditioned 
upon  it. 


To  handle  the  weights,  the  largest  of  the  weights  for  the  tier  immediately  below 
the  cost  category  was  selected  -  the  redeployment  cost  weight.  Use  has  the  highest 
relative  weight  and  was  varied  0.10,  providing  as  before  the  weight  vectors, 


"0.0077“ 

“0.01“ 

“0.0123“ 

0.2223 

,  w°  = 

0.29 

,  w  = 

0.3577 

0.77 

0.7 

0.63 

(133) 
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Figure  82.  Third  Group  Screening,  Cost  Portion  of  Value  Hierarchy. 

The  weights  for  the  subcategories  within  development,  redeployment,  and  use 
were  not  altered.  The  Cost  screening  experimental  design  is  illustrated  in  Table  34,  along 
with  the  results.  The  pattern  information  indicates  variables  levels  as  explained  above, 
with  a  variable  order  of  Development  variables  (Money,  Time,  Expertise,  and  Risk), 
Redeployment  variables  (Money,  Time,  Expertise,  and  Risk),  Use  variables  (Money, 
Time,  and  Expertise),  weight,  and  alternative.  Profde  is  not  included  as  a  variable  as  the 
data  is  identical  for  both  mission  profiles. 

The  least  squares  fit  of  the  data  is  provided  in  Figure  83  and  Table  35.  The  Cost 
Use  Time  is  highly  significant,  but  the  model  fit  is  mediocre,  with  R2  =  0.657  and 
Radj  =  0-523 .  Because  of  the  fit,  Cost  Redeployment  Time,  Cost  Use  Experience,  and 

Cost  Redeployment  Risk  will  be  retained  in  the  list  of  significant  variables  for  the 
moment. 
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Table  34.  Cost  Screening  Design  and  Response  for  ATR  Problem. 


Row 

Pattern 

u 

Row 

Pattern 

u 

1 

..++..++.+ — 

0.642 

27 

+.++-++-++-+ 

0.747 

2 

_J — 1 — 1 — 1 — 1 — 1 — U  4-0 

0.617 

28 

-4—1-  -1-4- 

0.445 

3 

0.612 

29 

+++-++-+-++-0 

0.477 

4 

- 

0.370 

30 

-+++ - + 

0.370 

5 

+  ++  + 

0.439 

31 

4 — | — | — | - 1—1—1- 

0.549 

6 

4-4-4-  4-  4-4-  O 

0.496 

32 

++  +  0 

0.522 

7 

++++-++++-+- 

0.703 

33 

++-+-+++--+++ 

0.552 

8 

4-  4 _ 1 _ 1 _ 1 _ U  4-fl 

0.617 

34 

_1_  -1—1-  4—1—1 _ U 

0.778 

9 

4— 1-  4-4-4 — 1-4- 

0.778 

35 

4-  4— i-  n 

0.312 

10 

++-++-+-+-+0 

0.443 

36 

-+-++-++++-+ 

0.703 

11 

4-  4 _ 1 _ 1 _ U  4 _ 1 _ UA 

0.546 

37 

4— 1-  4— 1— 1-  4—1—1- 

0.817 

12 

4-4— 1-4 1— 1-  4— 1— 1-  4- 

0.694 

38 

-i _ u  -i _ u  4-  n 

0.425 

13 

0.703 

39 

+-++—+-+++ 

0.456 

14 

.++...+..+.+0 

0.443 

40 

1 

+ 

1 

1 

+ 

1 

1 

1 

1 

1 

+ 

+ 

1 

0.664 

15 

++-+-+ — +.++ 

0.804 

41 

-+++-+ — ++-+0 

0.468 

16 

0.664 

42 

-+-+-+++-++-+ 

0.807 

17 

..++.+.-.+.0 

0.351 

43 

1 

1 

1 

1 

1 

1 

1 

1 

+ 

+ 

+ 

0.612  [ 

18 

0.756 

44 

— ++0 

0.475 

19 

-+-+++-++— 

0.705 

45 

0.837 

20 

-{- _ -f- _ _ 

0.642 

46 

_ -1— 1— 1— L-L 

0.778 

21 

+-+-+--++-++ 

0.815 

47 

+-+++— +—+-0 

0.438  | 

22 

0.526 

48 

0.553  i 

23 

+-+++ — ++++0 

0.444 

49 

00000000000- 

0.719 

24 

+-+++-+„++ 
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50 

000000000000 

0.500 

25 

+++++++-+- 

0.612 

51 

00000000000+ 
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26 
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Figure  83.  ATR  Cost  Model  Actual  versus  Predicted  Values. 
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Table  35.  ATR  Cost  Screening  Parameter  Estimates. 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob  >  |t| 

Intercept 

0.590 

0.0140 

42.27 

<.0001 

Development  Money(-l,l) 

-0.00393 

0.0143 

-0.28 

0.785 

Development  T ime(- 1,1) 

0.00171 

0.0143 

0.12 

0.906 

Development  Expertise^  1,1) 

0.0131 

0.0143 

0.91 

0.368 

Development  Risk(- 1,1) 

-0.00183 

0.0144 

-0.13 

0.899 

Redeployment  Money(- 1,1) 

0.0101 

0.0144 

0.70 

0.489 

Redeployment  Time(-l,l) 

0.0239 

0.0144 

1.66 

0.106 

Redeployment  Expertise^  1,1) 

0.0170 

0.0144 

1.18 

0.244 

Redeployment  Risk(- 1,1) 

0.0231 

0.0144 

1.60 

0.118 

Use  Money(-l,l) 

0.0169 

0.0144 

1.17 

0.249 

Use  Time(-l,l) 

0.0569 

0.0143 

3.97 

0.0003 

Use  Expertise^  1,1) 

0.0233 

0.0144 

1.62 

0.115 

Weights(-l,l) 

0.0261 

0.0144 

1.81 

0.0787 

ATR[ATR  331 

0.0805 

0.0196 

4.10 

0.0002 

ATR[ATR  551 

-0.114 

0.0196 

-5.83 

<.0001 

ATR[ATR  891 

0.0340 

0.0196 

1.73 

0.0924 

Fourth  Group  Screening.  The  Detection  Perfonnance  taxon  will  now  be 
examined.  This  decomposes  into  two  evaluation  measures,  False  Alann  Rate  given  a 
Probability  of  Detection  and  the  Probability  of  False  Alarm  given  a  Probability  of 
Detection.  (Alternatively,  it  could  have  been  structured  on  the  Required  Operating  Curve 
False  Alarm  Rate  and  the  Required  Operating  Curve  Probability  of  False  Alann.)  This  is 
depicted  in  Figure  84. 

The  weight  for  Probability  of  False  Alann  will  be  varied  ten  percent,  with  the 
weight  of  False  Alann  Rate  adjusted  according  so  that  the  pair  sum  to  one.  Coupled  with 
the  weights,  there  are  three  continuous  variables  plus  the  profile  and  alternative 
categorical  variables.  The  weight  vectors  are 


"0.3719” 

,  w°  = 

”0.429” 

”0.4861” 

0.6281_ 

0.571_ 

,  w~  = 

0.5139 
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Figure  84.  Fourth  Group  Screening,  Robustness  Portion  of  Value  Hierarchy. 


The  experimental  design  and  the  response  variables  are  contained  in  Table  36  and 
the  least  square  variable  significance  and  parameter  estimates  are  in  Table  37.  The 
pattern  infonnation  in  Table  36  indicates  variables  levels  as  explained  above,  with  a 
variable  order  of  FAR\Pd,  Pfa |  Pn ,  weight,  and  alternative.  Profile  data  is  again  identical 

for  both  mission  profiles.  As  seen  in  Table  37,  both  the  False  Alarm  Rate  and  the 
Probability  of  False  Alarm  were  found  to  be  significant. 


Table  36.  Detection  Performance  Screening  Experimental  Design  and  Response  for  ATR 

Problem. 


Row 

Pattern 

u 

Row 

Pattern 

u 

1 

— +- 

0.0850 

13 

-++++ 

0.626 

2 

+++- 

0.680 

14 

+— + 

0.486 

3 

0.317 

15 

— -+ 

0.383 

4 

0.0850 

16 

+++++ 

0.991 

5 

-+— 

0.260 

17 

....+ 

0.383 

6 

+-++- 

0.104 

18 

+++++ 

0.991 

7 

-++0 

0.0133 

19 

000- 

0.204 

8 

++--0 

0.919 

20 

000-0 

0.570 

9 

-+-0 

0.331 

21 

000-+ 

0.464 

10 

++-+0 

0.996 

22 

000+- 

0.468 

11 

-+-+0 

0.522 

23 

000+0 

0.562 

12 

+-+-0 

0.373 

24 

000++ 

0.480 
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Table  37.  ATR  Detection  Performance  Screening  Parameter  Estimates. 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob  >  |t| 

Intercept 

0.471 

0.0247 

19.02 

<.0001 

FAR(-1,1) 

0.1293 

0.0290 

4.46 

0.0003 

Pfa(-l,l) 

0.214 

0.0290 

7.38 

<.0001 

w(-l,l) 

-0.0207 

0.0290 

-0.71 

0.486 

ProfilefCIDl 

0.00384 

0.0250 

0.15 

0.880 

Profile[TSR] 

-0.00384 

0.0250 

-0.15 

0.880 

ATR[ATR  331 

-0.195 

0.0350 

-5.58 

<.0001 

ATR[ATR  55] 

0.0653 

0.0350 

1.86 

0.0796 

ATR[ATR  891 

0.130 

0.0350 

3.72 

0.0017 

Fifth  Group  Screening.  Examining  Classification  Ability  (see  Figure  85),  the 
weight  vectors  are  established  by  selecting  the  larger  weight,  that  of  Classification  by 


Class,  to  vary  by  ten  percent. 


"0.4734” 

,  w°  = 

”0.526” 

”0.5786” 

0.5216 

0.474 

,  w  = 

0.4214 
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Figure  85.  Fifth  Group  Screening,  Classification  Ability  Portion  of  Value  Hierarchy. 


Table  38  contains  the  experimental  design  information  and  the  response.  The 
pattern  infonnation  indicates  variables  levels  as  explained  above,  with  a  variable  order  of 
PID ,  Pcc ,  weight,  profile,  and  alternative.  The  coefficients  and  significance  statistics  are 
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Table  38.  Classification  Ability  Screening  Design  and  Response  for  ATR  Problem. 


Row 

Pattern 

u 

Row 

Pattern 

u 

1 

— +- 

0.00106 

13 

-++++ 

0.552 

2 

+++- 

1.000 

14 

+— + 

0.529 

3 

0.684 

15 

— -+ 

0.0436 

4 

0.508 

16 

+++++ 

0.967 

5 

-+— 
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17 

— -+ 

0.0436  1 
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+-++- 
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18 

+++++ 

0.967 

7 

-++0 

0.0150 

19 

000- 

0.811  | 

8 

++--0 

0.995 

20 

000-0 

0.769 

9 

-+-0 

0.110 

21 

000-+ 

0.655  1 

10 

++-+0 
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22 

000+- 

0.293 

11 

-+-+0 

0.484 

23 

000+0 

0.557 

12 

+-+-0 

0.455 

24 

000++ 

0.374 

contained  in  Table  39.  Both  Classify  by  Type  and  Classify  by  Class  evaluation  measures 
are  highly  significant. 


Table  39.  ATR  Classification  Ability  Screening  Parameter  Estimates. 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob  >  |t| 

Intercept 

0.534 

0.0204 

26.12 

<.0001 

Type(-l,l) 

0.215 

0.0240 

8.97 

<.0001 

Class(-l,l) 

0.267 

0.0239 

11.2 

<.0001 

w(-l,l) 

0.00705 

0.0239 

0.29 

0.772 

Profile  [CID1 

0.0649 

0.0207 

3.14 

0.0060 

Profile  [ISR1 

-0.0649 

0.0207 

-3.14 

0.0060 

ATR[ATR  331 

0.00477 

0.0289 

0.16 

0.871 

ATR[ATR  551 

0.0129 

0.0289 

0.45 

0.661 

ATR1ATR  891 

-0.0177 

0.0289 

-0.61 

0.550 

Sixth  Group  Screening.  Finally  the  Self  Assessed  Accuracy  category  is 
considered.  This  is  depicted  in  Figure  86.  The  data  and  both  value  and  utility  functions 
are  identical  for  the  three  evaluation  measures  in  this  category.  The  input  data  are  all 
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Self  Assessed 
Accuracy 
(0.14) 


Det  Accuracy 
[Es-Pd] 
(0.333) 


Type  Accuracy 
[Es-Pid] 
(0.333) 


Class  Accuracy 
[Es-Pcc] 
(0.333) 


Figure  86.  Sixth  Group  Screening,  Self-Assessed  Accuracy  Portion  of  Value  Hierarchy. 

unity,  providing  value  and  utility  of  zero,  as  this  evaluation  measure  was  not  part  of  the 
ATR  CS  testing.  So  testing  this  category  as  above  is  useless,  as  all  combinations  of 
weights  and  utility  exponential  values  produce  a  utility  of  zero.  Clearly  the  Self  Assessed 
Accuracy  measures  are  not  significant. 

Prioritization  of  Evaluation  Measures.  The  ordered  coefficients  are  contained 
in  Table  40.  They  were  detennined  by  multiplying  coefficients  by  the  coefficient  of 
parent  groups  to  provide  a  global  comparison.  These  provide  a  reasonable  order  in  which 
single  dimensional  value  functions  should  be  converted  to  single  dimensional  utility 
functions  in  the  hybrid  model. 

The  value  hierarchy  reduced  to  those  evaluation  measures  that  were  detennined  to 
be  significant  for  a  =  0. 10  during  the  screening  process  is  depicted  in  Figure  87. 
Examining  only  these  evaluation  measures  produces  the  subset  of  model  coefficients 
listed  in  Table  41.  The  entries  of  Table  41  are  identical  to  the  top  eleven  entries  of  Table 
40  except  that  Declaration  Ability  and  Classification  Robustness  do  not  appear  in  Table 
41.  These  evaluation  measures  were  not  found  to  be  significant  during  the  screening. 
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Table  40.  Rank  Ordered,  by  Absolute  Value,  Model  Coefficients. 


Evaluation  Measure 

Coefficient 

Employment  Concept 

0.0738 

Detection  Robustness 

0.0146 

Probability  of  False  Alann 

0.0123 

Classification  by  Class 

0.0122 

Classification  by  Type 

0.00980 

Identification  Robustness 

0.00907 

False  Alarm  Rate 

0.00746 

Declaration  Ability 

0.00211 

Cost  Use  Time 

0.000794 

Classification  Robustness 

0.000729 

Cost  Redeployment  Time 

0.000334 

Cost  Use  Expertise 

0.000324 

Cost  Redeployment  Risk 

0.000322 

Cost  Redeployment  Expertise 

0.000238 

Cost  Use  Money 

0.000235 

Cost  Development  Expertise 

0.000183 

Redeployment  Money 

0.000141 

Cost  Development  Money 

-5.48  x  10'5 

Cost  Development  Risk 

-2.55  x  10'5 

Cost  Development  Time 

2.38  x  10  5 

Es  Pd 

0 

Es  Pid 

0 

Es  Pec 

0 

They  are  in  the  eighth  and  tenth  positions  of  Table  40.  We  conclude  that  employing  the 
significance  as  a  screening  tool  is  reasonable. 

Employment  of  the  Hybrid  Value-Utility  Model.  The  hybrid  utility  algorithm 
was  applied,  using  the  priority  sequence  of  variables  presented  in  Table  41.  Iteration  zero 
employs  the  value  model.  Each  succeeding  iteration  provides  a  hybrid  model  with  the 
next  highest  priority  evaluation  measure  value  function  replaced  by  the  utility  function. 
The  results  for  the  mission  CID  profile  are  presented  in  Table  42  and  Table  43  contains 


206 


Figure  87.  Value  Hierarchy  Reduced  to  Significant  Taxa.  The  parenthetical 
numbers  indicate  the  estimated  coefficients.  Taxa  significant  at  alpha  equal  to  0.1  and 
not  when  alpha  equal  0.05  are  indicated  with  an  asterisk. 


Table  41.  Significant  Rank  Ordered  Model  Coefficients.  Those  significant  at  an  alpha 
equal  to  0. 1  and  not  when  alpha  equal  0.05  are  indicated  with  an  asterisk. 


Evaluation  Measure 

Coefficient 

Priority 

Concept 

0.0738 

1 

Detection  Robustness 

0.0146 

2 

Probability  of  False  Alann 

0.0123 

3 

Classification  by  Class 

0.0122 

4 

Classification  by  Type 

0.00980 

5 

Identification  Robustness 

0.00907 

6 

False  Alarm  Rate 

0.00746 

7 

Cost  Use  Time 

0.000794 

8 

Cost  Redeployment  Time* 

0.000334 

9 

Cost  Use  Expertise* 

0.000324 

10 

Cost  Redeployment  Risk* 

0.000322 

11 

the  results  for  the  ISR  profile.  The  pure  utility  model  results  are  also  provided  in  the 
respective  tables.  Results  for  conversion  of  all  variables,  not  only  the  significant  ones, 
from  value  to  utility  are  provided  in  Appendix  E. 
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Table  42.  CID  Profile  Value,  Utility,  and  Hybrid  Utility  Results,  Significant  Evaluation 
Measures.  Those  significant  at  an  alpha  equal  to  0. 1  and  not  when  alpha  equal  0.05  are 

indicated  with  an  asterisk. 


Notes 

ATR33 

ATR55 

ATR  89 

Goal 

Utility  Model 

0.572 

0.507 

0.518 

Iteration 

0 

Value  Model 

0.509 

0.556 

0.525 

1 

Employment  Concept 

0.464 

0.511 

0.480 

2 

Detection  Robustness 

0.455 

0.471 

0.442 

3 

Probability  of  False  Alarm 

0.529 

0.500 

0.512 

4 

Classification  by  Class 

0.536 

0.509 

0.524 

5 

Classification  by  Type 

0.538 

0.511 

0.528 

6 

Identification  Robustness 

0.529 

0.505 

0.509 

7 

False  Alarm  Rate 

0.569 

0.506 

0.510 

8 

Cost  Use  Time 

0.569 

0.506 

0.513 

9 

Cost  Redeployment  Time* 

0.570 

0.507 

0.513 

10 

Cost  Use  Expertise* 

0.570 

0.507 

0.513 

11 

Cost  Redeployment  Risk* 

0.570 

0.504 

0.516 

Table  43.  ISR  Profile  Value,  Utility,  and  Hybrid  Utility  Results,  Significant  Evaluation 
Measures.  Those  significant  at  an  alpha  equal  to  0. 1  and  not  when  alpha  equal  0.05  are 

indicated  with  an  asterisk. 


Notes 

ATR  33 

ATR  55 

ATR  89 

Goal 

Utility  Model 

0.415 

0.455 

0.439 

Iteration 

0 

Value  Model 

0.497 

0.531 

0.500 

1 

Employment  Concept 

0.452 

0.486 

0.452 

2 

Detection  Robustness 

0.443 

0.446 

0.414 

3 

Probability  of  False  Alarm 

0.371 

0.415 

0.383 

4 

Classification  by  Class 

0.392 

0.430 

0.403 

5 

Classification  by  Type 

0.405 

0.435 

0.413 

6 

Identification  Robustness 

0.395 

0.429 

0.394 

7 

False  Alann  Rate 

0.412 

0.455 

0.432 

8 

Cost  Use  Time 

0.412 

0.455 

0.434 

9 

Cost  Redeployment  Time* 

0.413 

0.455 

0.434 

10 

Cost  Use  Expertise* 

0.413 

0.455 

0.434 

11 

Cost  Redeployment  Risk* 

0.413 

0.452 

0.434 
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The  hybrid  utility  model  results  are  shown  graphically  in  Figure  88  for  the  ATR 
33  alternative.  Other  alternatives  behaved  similarly  and  are  omitted  for  clarity.  All  three 
alternatives  converge  towards  the  utility  model  after  an  initial  period  of  instability.  This 
is  more  readily  observed  in  Figure  89  through  Figure  92,  where  the  differences  between 
iterative  hybrid  model  scores  and  the  distance  from  the  true  utility  model  for  the  two 
mission  profiles  are  illustrated.  The  decreasing  differences  between  iterations  and  the 
decreasing  distance  from  the  utility  model  are  obvious.  The  latter  measure  is  a  measure 
of  the  error  of  the  hybrid  model  compared  to  the  true  utility  model.  After  the  initial 
period  this  error  remains  small. 


♦  ATR  33  Hybrid  (CID) 

- ATR  33  Utility  (CID) 

A  ATR  33  Hybrid  (ISR) 

- ATR  33  Utility  (ISR) 


Iteration 


Figure  88.  ATR  Hybrid  Utility  Iterations  for  ATR  33. 
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□  ATR  33 
■  ATR  55 

□  ATR  89 


Figure  89.  Hybrid  Utility  Model  Difference  Between  Successive  Iterations,  CID  Profile. 


□  ATR  33 
■  ATR  55 


Figure  90.  Hybrid  Utility  Model  Difference  Between  Successive  Iterations,  ISR  Profile. 
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0.09 


□  ATR  33 
■  ATR  55 

□  ATR  89 


-0.09 


Iteration 


Figure  91.  Hybrid  Utility  Model  Difference  Between  Iteration  and  True  Utility  Model, 

CID  Profile. 


□  ATR  33 
■  ATR  55 


Figure  92.  Hybrid  Utility  Model  Difference  Between  Iteration  and  True  Utility  Model, 

ISR  Profile. 
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To  understand  the  behavior  of  the  hybrid  model  as  it  is  modified  during  each 
iteration,  we  examine  the  initial  step  for  the  ATR  33  alternative  (see  Table  42  and  Figure 
88).  The  complete  value  model  had  an  overall  value  of  0.509  for  this  alternative.  When 
the  Employment  Concept  was  converted  to  the  single  dimensional  utility  function,  the 
hybrid  utility  became  0.464.  The  Employment  Concept  rating  for  the  ATR  33  alternative 
(indeed,  all  alternatives)  was  poorly  defined.  The  corresponding  single  dimensional 
value  is  0.4.  The  single  dimensional  utility  for  the  poorly  defined  rating  is  0. 1,  so  the 
difference  between  the  preference  functions  is  0.3.  When  this  difference  is  multiplied  by 
the  relative  weight  for  the  Employment  Concept  evaluation  measure,  0.15,  the  result  is 
0.45,  the  difference  between  the  results  obtained  in  the  first  iteration  compared  to  the 
starting  value  model. 

It  is  interesting  to  note  in  Figure  88  that  ATR  33  converges  to  the  true  utility 
model  strictly  from  below  for  the  CID  profile,  although  the  direction  of  movement  was 
not  identical  for  each  iteration,  while  under  the  ISR  case  ATR  33  oscillated  about  the  true 
utility  value.  In  no  case  was  the  movement  strictly  towards  the  true  utility,  as  would  be 
expected  if  the  decision  maker  was  strictly  relatively  risk  averse  (seeking). 

Once  the  significant  evaluation  measures,  at  a  =  0.05 ,  were  converted  from  value 
to  utility  functions  the  hybrid  function  provided  estimates  that  differed  at  most  0.003 
from  the  pure  utility  function  result.  The  evaluation  measures  that  became  significant 
between  a  =  0.05  and  a  =  0.1  provided  no  improved  estimate. 

We  conclude  that  for  the  example  the  evaluation  measures  determined  to  be 
significant  through  response  surface  methodology  techniques  provided  an  adequate  set  of 
variables  to  permit  creation  of  an  accurate  hybrid  model.  The  hybrid  model  may  be  used 
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to  represent  the  utility  model.  The  number  of  preference  functions  required  to  be  elicited 


from  the  decision  maker  was  reduced  from  26  to  eight  for  a  =  0.05  . 

Further,  the  rank  ordering  by  regression  coefficient  was  successful  in  providing 
successive  hybrid  estimates  that  in  general  approached  the  true  utility  value,  each  with 
generally  reduced  difference  (<5))  from  the  previous  estimate. 


Hybrid  Value-Utility  Summary 

Research  Summary.  This  portion  of  the  research  permits  several  conclusions. 
The  first  objective  under  this  area  of  research  was  to  establish  when  value  and  utility 
functions  differed  significantly.  It  was  demonstrated  that  application  of  response  surface 
methodology  to  the  preference  functions  provides  key  infonnation  regarding  the  behavior 
of  the  utility  model.  This  provides  information  that  indicates  for  which  evaluation 
measures  value  and  utility  differ  significantly.  This  is  a  new  contribution  and  is  useful  to 
identify  those  functions  where  discrimination  must  be  made  between  value  and  utility.  It 
would  also  be  useful  to  identify  where  a  linear  preference  function,  employed  in  the 
absence  of  data,  is  adequate.  When  adequate,  then  no  effort  need  be  dedicated  to 
determining  the  actual  function. 

The  second  objective  was  to  develop  an  algorithm  that  provides  an  adequate 
hybrid  value-utility  model.  The  algorithm  was  successful  in  providing  the  hybrid  model. 

The  hybrid  utility  model,  U ,  provided  an  adequate  representation  of  the  true  utility 
model,  U  .  Elicitation  of  the  hybrid  utility  model  is  more  efficient  with  respect  to 
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decision  maker  participation  than  employing  the  strict  utility  model.  Under  conditions  of 
uncertainty  and  risk,  the  value  model  may  provide  differing  results  than  the  utility  model, 
as  was  the  case  here,  and  so  is  not  acceptable.  This  agrees  with  the  results  of  Chapter  IV, 
that  value  and  utility  functions  often  differ  significantly. 

Potential  Future  Work.  Future  research  could  examine  improvements  in  the 
stopping  criteria  of  the  algorithm.  Earlier  stopping  of  the  algorithm,  within  acceptable 
tolerances,  provides  returns  on  the  efficiency  with  respect  to  the  decision  maker. 

The  method  of  incorporation  of  the  alternative  in  the  experimental  design  may  be 
an  area  where  future  research  would  increase  the  efficiency  of  the  hybrid  model 
methodology.  Treating  the  alternative  as  a  categorical  variable  in  the  design  greatly 
limits  available  designs.  If  the  set  of  alternatives  may  be  reduced  during  the  examination 
of  the  preference  functions  potential  significance,  many  more  experimental  designs  are 
available.  Clearly  if  a  single  alternative  may  be  used  for  this  step,  the  algorithm  becomes 
much  more  efficient.  Less  grouping  of  variables  would  be  required,  meaning  that  fewer 
successive  group  screening  steps  would  be  required.  Determination  of  sufficient 
conditions  when  this  simplification  is  acceptable  would  be  beneficial. 

Rank  statistical  methods  may  provide  an  additional  stopping  criterion.  If  single 
dimensional  substitutions  are  made  in  decreasing  order  of  the  strength  of  the  subject’s 
risk  attitudes,  then  each  iteration  will  have  decreasing  effects  of  the  hybridization 
process.  Each  iteration  i  provides  a  set  R>i]  of  the  estimates  of  the  utility  for  each 
alternative.  The  Mann-Whitney  (or  Wilcoxon)  test  provides  a  means  of  determining  if 
the  rankings  of  each  iteration  are  drawn  from  the  same  population  (Conover,  197 1 :  213  — 
239).  If  consecutive  rankings  are  not  from  significantly  different  populations,  then  the 
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model  produced  is  not  significantly  different  from  its  predecessor.  It  is  possible  that  the 
algorithm  may  be  terminated,  although  likely  certain  conditions  may  be  necessary. 
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VI.  Summary 


General 

The  research  had  two  major  objectives,  both  pertaining  to  the  relationship  of 
value  and  utility  preference  functions  in  decision  analysis.  The  first  was  to  examine  this 
relationship  itself  between  value  and  utility  functions.  Possible  relationships  between  the 
functions,  each  with  underlying  theoretical  implications,  were  examined.  Clarification  of 
the  relationships  provides  insight  into  risk  attitudes  as  well  as  facilitates  representation  of 
empirical  data  with  functional  forms.  The  second  was  to  improve  the  efficiency  of 
decision  analysis  by  detennining  when  the  distinctions  between  value  and  utility 
preference  functions  were  significant.  This  chapter  will  summarize  the  research, 
highlight  the  original  contributions,  and  identify  possible  future  work.  Detailed 
explanation  of  the  results  accompanies  the  presentation  of  the  methodology,  analysis,  and 
results  in  Chapters  IV  and  V. 


Relationship  of  Value  and  Utility  Preference  Functions 

Functional  Relationship  of  Single  Dimensional  Value  and  Utility  Functions. 

The  analysis  of  the  data  for  both  the  tactical  scenario  and  the  information  operations 
study  clearly  show  that  the  value  and  utility  preference  functions  differ  significantly  and 
so  are  different  constructs.  For  the  tactical  scenario,  the  linear  model  of  v(.  ( xt )  =  ui  (jc(. ) , 
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i.e.,  a  linear  transformation  of  value  into  utility,  failed  to  achieve  the  best  fit  for  any 
subject  on  any  evaluation  dimension.  It  achieved  an  acceptable  fit,  within  the  elicitation 
error,  less  than  five  percent  of  the  time.  This  was  a  poorer  perfonnance  than  any  other 
model.  For  the  information  operations  analysis,  the  linear  model  also  failed  to  achieve 
the  best  fit  for  any  evaluation  dimension.  The  linear  model  had  an  acceptable  fit  for  only 
one -third  of  the  evaluation  measures,  which  was  again,  along  with  the  logarithmic  model, 
the  poorest  perfonnance. 

The  sigmoid  model,  a  sigmoid  transformation  of  value  into  utility,  perfonned 
well.  It  provided  the  best  fit  in  over  90  percent  of  the  cases  as  determined  using  WRMS 
(84  percent  for  RMS).  The  other  functions  were  very  poor  performers,  typically  fitting 
best  in  only  a  few  cases.  The  best  of  these  other  functions  was  the  exponential,  which  is 
typically  the  relationship  of  choice  of  the  decision  analyst.  The  exponential  and  power 
functions  achieved  an  acceptable  fit  (as  opposed  to  best  fit)  about  eleven  percent  of  the 
time,  logarithmic  about  seven,  and  linear  less  than  in  five  percent  of  the  cases.  This 
supports  the  hypothesis  that  the  value  and  utility  functions  are  different  constructs  and 
different  within  a  professional  population  of  subjects.  Further,  it  indicates  that 
approximating  a  utility  function  with  the  corresponding  value  function  is  a  poor 
approximation  in  the  absence  of  other  information. 

Characteristics  And  Performance  Of  Various  Common  Decision  Analysis 
Elicitation  Methodologies.  Initially  it  is  important  to  consider  if  the  distinction  between 
value  and  utility  are  inconsequential.  Several  researchers  have  reported,  as  reviewed  in 
Chapter  II,  that  elicitation  error  often  exceeds  the  differences  between  value  and  utility 
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functions.  If  so,  then  value  and  utility  elicitation  methods  would  be  interchangeable  in 
application. 

No  previous  estimates  of  elicitation  error  were  located,  although  Keller  (1985b: 
479)  had  proposed  an  ad  hoc  criterion  of  RMSE  less  than  or  equal  to  0.05  for  acceptance 
of  a  model  fit  to  elicited  data.  This  research  includes  an  estimate  of  elicitation  error 
between  the  certainty-equivalent  and  probability-equivalent  methods  of  utility  function 
elicitation.  This  provides  the  criterion  of  0. 107  for  WRMSE  (or  0. 120  for  RMSE  if 
WRMSE  is  not  used)  for  acceptable  fit  of  utility  functions.  For  the  tactical  scenario 
elicitations,  which  involved  standard  elicitation  methodologies,  the  disparity  between  the 
value  function  and  the  mean  utility  function  was  0.191.  These  differences  are  statistically 
significant.  This  indicates  that  value -based  and  utility-based  approaches  are  not 
equivalent,  and  each  should  be  used  only  in  the  proper  context. 

The  two  single  dimensional  utility  elicitation  procedures  examined  in  the  tactical 
scenario,  the  probability  and  certainty  equivalent  approaches,  were  found  to  produce 
strategically  equivalent  results.  Employing  either  method  alone  would  produce  results 
that  did  not  significantly  differ,  although  the  p  critical  value  was  0.055.  This  suggests 
that  either  method  may  be  employed,  but  eliciting  both  and  using  an  average  of  the  two 
may  be  advisable  in  critical  applications.  Subjects  involved  in  the  tactical  scenario 
elicitations  indicated  that  they  were  more  comfortable  providing  answers  to  the 
probability  equivalent  approach  rather  than  the  certainty  equivalent  method.  Single 
dimensional  value  and  utility  functions  were  clearly  different  constructs.  This  clearly 
supports  employing  value  functions  only  under  conditions  of  certainty,  and  using  utility 
functions  otherwise. 
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The  multiattribute  utility  approach  of  Kirkwood  (1997b:  155  -  166)  was  found  to 
provide  results  that  were  not  strategically  equivalent.  This  is  discussed  further  below. 

Improving  The  Ability  to  Compare  Between  Preference  Functions.  Previous 
work  employed  the  root  mean  square  error  (RMSE)  as  the  method  for  comparing  two 
preference  functions.  A  weighted  RMSE,  WRMSE,  which  gives  more  weight  to  the 
differences  further  from  the  center  of  the  independent  variable  midpoint,  is  more 
appealing.  Differences  of  equal  magnitude  between  two  functions  that  occur  at  different 
distances  from  the  center  point  occur  under  differing  constraints.  The  functions  are  most 
constrained  at  the  endpoints  of  the  domain,  so  equal  differences  at  different  distances 
form  the  midpoint  should  be  accorded  different  relative  weighting.  Such  a  WRMSE  was 
developed  in  this  work,  and  is  provided  in  equation  (58)  in  Chapter  IV. 

As  utility  functions  were  elicited  employing  two  different,  but  compatible, 
methods,  differences  in  these  two  elicited  functions  were  assumed  to  capture  the  error  of 
the  elicitation  process.  Analysis  of  this  error  provided  the  criteria  of  0. 120  for  RMSE  and 
0. 107  for  WRMSE  for  acceptable  fit  of  utility  functions.  Differences  between  two 
functions  below  these  criteria  are  not  statistically  significant  different  from  the  elicitation 
error  (for  these  data),  at  the  a  =  0.05  level.  These  criteria  are  a  new  contribution  to  the 
literature.  It  is  recommended  that  the  WRMSE  be  employed  to  detect  significant 
differences  between  preference  functions,  with  the  criterion  of  0.107. 

Examining  The  Subject’s  Risk  Attitudes.  The  curve  shape  of  the  u  (v  (jc)) 

function  (transformation  function  of  value  to  utility)  provides  insight  into  a  decision 
maker’s  risk  attitudes,  as  does  which  mathematical  model  best  fits  the  elicited  data.  In 
the  tactical  movement  problem,  the  concave  curve  was  exhibited  most  frequently  (39  of 
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80  curves)  and  the  curves  were  S-shaped  in  32  of  the  cases.  The  sigmoid  fit  best  for 
more  than  ninety  percent  of  the  evaluation  measures.  The  exponential  and  logarithmic 
models  fit  for  less  than  five  percent  of  the  cases.  The  sigmoid  model  (being  more 
flexible)  fit  best  in  almost  sixty  percent  of  the  cases;  the  exponential  in  more  than  forty. 
For  the  infonnation  operations  analysis,  the  sigmoid  function  provided  the  best  fit  in 
seven  cases  and  the  exponential  in  the  remaining  five.  Of  the  twelve  continuous 
evaluation  measures,  the  curve  shapes  of  the  assessed  utility  functions  of  the  subjects 
were  not  strongly  risk  averse:  two  were  linear  or  near  linear,  four  were  S-shaped,  four 
were  convex,  and  two  were  concave. 

This  lack  of  dominance  of  concave  utility  functions  and  the  fit  of  the  sigmoid 
function  suggests  that  risk  attitudes  are  more  complex  than  the  constant  risk  attitudes 
usually  assumed.  The  usual  assumption  is  constant  risk  aversion.  If  a  constant  risk 
attitude  is  to  be  assumed,  constant  risk  aversion  was  the  best  model,  which  is  supported 
by  the  high  number  of  fits  of  the  exponential  function.  However  the  assumption  of 
constant  risk  aversion  for  military  decision  making  is  not  supported  by  these  data. 

Consistent  risk  attitude  was  defined  as  the  condition  that  the  decision  maker 
maintains  risk  attitudes  on  all  evaluation  measures  that  are  consistent  with  respect  to 
being  generally  risk  averse,  risk  neutral,  or  risk  seeking.  If  this  condition  is  present,  then 
the  decision  maker  is  risk  averse  (neutral)  (seeking)  on  all  evaluation  measures  if  she  is 
risk  averse  (neutral)  (seeking)  on  any  single  evaluation  measure.  The  subjects 
considering  the  tactical  scenario  did  not  demonstrate  consistent  risk  attitudes.  A  new 
metric,  A  ,  was  created  and  used  to  quantify  the  variability  of  the  subjects’  risk  attitudes. 
This  metric  combines  measures  of  local  and  global  risk  attitudes  on  an  evaluation 
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measure,  a  new  contribution.  It  appears  that  the  subjects’  risk  attitudes  varied  widely 
between  evaluation  measures.  Such  variance  contraindicates  the  employment  of  the 
multiattribute  utility  approach  of  Kirkwood  (1997b),  which  assumes  risk  consistency,  for 
military  decision  making. 

Learning  effects  were  studied.  It  was  hypothesized  that  elicitation  of  single 
dimensional  value  functions  would  clarify  the  subjects’  thought.  So  elicitation  order  of 
value-utility-utility  should  see  lower  elicitation  error  than  utility-value -utility.  No 
learning  effects  were  observed. 

Improving  The  Metric  For  Assessing  Risk  Aversion.  This  research  improved 
the  ability  to  examine  subjects’  risk  attitudes.  The  measures R(f)  and  V ( f )  were 

developed  and  provide  more  discrimination  that  the  previous  metrics  -  curve  shape  and 
preference  area.  These  metrics  were  also  employed  in  the  search  for  consistent  risk 
attitudes  for  subjects,  measured  by  A  ,  discussed  above. 


Hybrid  Value-Utility  Decision  Analysis  Models 

The  second  major  portion  of  the  research  dealt  with  the  sensitivity  of  the  decision 
analysis  model  to  the  preference  function,  and  how  to  take  advantage  of  lack  of 
sensitivity.  The  work  of  Chapter  V  is  summarized  here. 

Establishing  When  Value-Utility  Differences  Are  Significant.  Response 
surface  methodology  was  successfully  applied  to  the  problem  of  detennining  when  the 
model  is  sensitive  to  the  preference  functions.  Response  surface  methodology  was  used 
to  examine  the  margins  of  an  envelope  of  reasonable  curves  for  uj  (v;  (x, )) .  When  the 
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function  failed  to  be  significant,  then  the  true  function  was  assumed  to  be  not  significant 
for  evaluation  measure  i. 


Developing  an  Algorithm  for  Employment  of  a  Hybrid  Value-Utility  Model. 

An  algorithm  was  created  that  was  successful  in  providing  the  hybrid  model.  The 

hybrid  multiattribute  utility  model,  U  ,  provided  at  adequate  representation  of  the  true 
multiattribute  utility  model,  U  .  Elicitation  of  the  hybrid  utility  model  is  more  efficient 
with  respect  to  decision  maker  participation  than  employing  the  strict  utility  model, 
especially  in  cases  where  there  are  a  large  number  of  evaluation  measures  or  cases  where 
several  of  the  measures  require  an  inordinate  amount  of  time  to  elicit  additional  utility 
information.  Such  efficiency  promotes  use  of  decision  analysis  and  encourages  sound 
decision  making. 


Original  Contributions 

The  results  discussed  above  include  original  contributions  to  the  literature.  This 
study  extended  value-utility  studies  into  multidimensional  cases  employing  military 
professionals  as  subjects.  An  improved  measure  of  differences  between  preference 
functions,  WRMSE,  was  created.  Acceptance  criteria  for  adequate  model  fit,  based  on 
elicitation  error,  were  established  for  WRMSE  and  RMSE.  The  concept  of  a  hybrid 
value-utility  decision  analysis  model  was  first  advanced.  The  sensitivity  of  decision 
analysis  models  to  perturbation  of  the  preference  functions  was  examined.  This  involved 
the  first  use  of  RSM  on  the  preference  functions.  This  infonnation  was  used  to  establish 
an  efficient  algorithm  for  creation  of  hybrid  value-utility  models. 
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Future  Research 


As  with  any  work,  additional  research  topics  emerge.  The  examination  of  value 
and  utility  function  relationships  presented  in  Chapter  IV  could  be  extended.  Simply 
enlarging  the  sample  size  would  improve  the  clarity  of  the  elicitation  error  infonnation. 
Repeating  elicitations  but  limiting  the  domain  for  the  utility  function  elicitation  to  the 
region  where  v;  (x;.)  =  a,  a  e  (0,l) ,  would  provide  stronger  evidence  for  the  usefulness  of 

the  sigmoid  function  if  it  continued  to  fit  well.  Repeating  the  work  with  military 
professionals  but  employing  a  different  scenario  would  strengthen  insight  that  the 
preference  function  results  are  not  scenario-specific. 

Learning  effects  discovered  during  the  course  of  this  research  were  significant, 
but  counterintuitive.  Examination  of  this  phenomenon  may  provide  useful  insights  into 
risk  attitudes.  Other  weighting  functions  for  the  weighted  root  mean  square  calculations 
may  offer  improved  metrics.  Application  of  WRMSE  to  discrete  functions  was  not 
addressed  in  this  research,  and  so  remains  an  open  question. 

Examination  of  the  risk  attitude  metrics  developed  in  this  work  for  non-military 
decision  making  may  show  improved  opportunity  for  use.  As  risk  attitude  consistency  is 
usually  assumed,  it  may  be  present  in  other  groups  of  decision  makers.  If  so,  the  metric 
A  may  prove  useful.  If  some  professional  group  is  found  to  possess  consistent  risk 
attitudes,  then  risk  attitude  information  for  a  subset  of  evaluation  dimensions  and  A  may 
be  used  to  estimate  risk  attitudes  on  other  dimensions.  Such  an  approach  could  provide 
improved  efficiency  for  problems  with  many  evaluation  dimensions. 
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Improved  stopping  criteria  for  the  hybridization  algorithm  would  improve 
efficiency.  Rank  order  statistics  may  provide  such  a  benefit,  particularly  if  conclusions 
about  strict  monotonicity  of  preference  functions  could  be  elicited  from  the  decision 
maker. 


Conclusion 

This  work  is  the  first  to  enter  the  literature  regarding  comparison  of  value  and 
utility  functions  for  decision  analysis  examining  multidimensional  data  from  military 
professionals  as  subjects.  This  research  supports  the  hypothesis  that  single  dimensional 
value  and  utility  functions  are  not  equivalent.  This  conclusion  was  confirmed  for  two 
separate  military  decision  making  situations.  No  single  functional  relationship  defined 
the  relationship  between  value  and  utility  functions.  Certainty  equivalent  and  probability 
equivalent  elicitation  methodologies  provided  equivalent  results.  Value  and  utility 
functions  differed  significantly  more  than  can  be  explained  by  elicitation  error. 
Aggregating  single  dimensional  utilities  into  a  multiattribute  model  and  the  Kirkwood 
(1997b)  multiattribute  utility  produce  results  that  are  not  strategically  equivalent. 

The  research  demonstrated  that  response  surface  methodology  may  be  applied  to 
the  preference  function.  The  potential  significance  of  a  preference  function  may  be 
employed  to  construct  a  hybrid  value-utility  model  that  facilitates  more  efficient 
elicitation  of  information  from  the  decision  maker.  This  promotes  use  of  decision 
analysis  techniques  and  therefore  improves  decision  making  for  large  multidimensional 
problems. 
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While  value  and  utility  are  differing  constructs,  prudent  exploitation  of  their 
differences  and  similarities  permits  more  efficient  use  of  the  decision  maker’s  time.  This 
work  emphasizes  that  the  distinction  between  value  and  utility  must  not  be  ignored.  But 
the  lack  of  strategic  differences  for  specific  cases  may  be  identified  and  exploited. 
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Appendix  A.  Glossary  of  Technical  Terms 


ALLAIS  PARADOX  -  Allais  constructed  two  sequential  lottery  choices,  that  most 
subjects  made  alternative  selections  in  each  that  are  mutually  incompatible  without 
axiomatic  violations  of  expected  utility.  Consider  four  lotteries.  The  notation  indicates 
the  payoff  then,  parenthetically,  the  associated  probability. 

L,  :  [$30,000  (0.33);  $25,000  (0.66>,  $0  (0.0l)] 

L2  :  [$25,000  (l)] 

L 3  :  [$30,000  (0.33);  $0  (0.67)] 

L4  :  [$25,000  (0.34);  $0  (0.66)] 

Initially  the  subject  must  choose  between  Li  and  L2 .  Most  subjects  choose  Ln .  Then  the 
same  subject  who  chose  L2  generally  chooses  L3  when  presented  with  a  choice  between 
L3  and  L4 .  This  combination  of  choices  violates  expected  utility.  Counter  arguments 
included  bounded  rationality,  FRAMING,  and  DYNAMIC  CONSISTENCY.  (Biswas, 
1997:5-6) 

ALTERNATIVE  -  One  of  several  courses  of  action  that  may  be  chosen  (Decision 
Analysis  Society,  November  3,  2000). 

BETWEENESS  AXIOM.  -  “If  someone  is  indifferent  between  two  lotteries  Ll  and  L2 , 
then  she  is  indifferent  between  either  of  these  lotteries  and  the  compound  lottery 
\Ll  (a),  L2  (l  -  a)] ,”  where  a  is  the  associated  probability  of  lottery  Lx .  Normally  this 
axiom  is  employed  in  conjunction  with  the  WEAK  INDEPENDENCE  AXIOM.  (Biswas, 
1997:  12)  See  INDEPENDENCE  AXIOM. 

CASH  EQUIVALENT  -  See  CERTAINTY  EQUIVALENT. 

CERTAIN  EQUIVALENT  -  See  CERTAINTY  EQUIVALENT. 

CERTAINTY  -  Condition  under  which  decision  are  made  with  known  detenninistic 
alternatives. 

CERTAINTY  EQUIVALENT  -  The  quantity  at  which  a  decision  maker  is  indifferent 
between  two  alternatives,  one  being  that  amount  under  certainty  and  the  other  an 
uncertain  situation  where  the  possibility  exists  of  either  a  greater  of  lesser  payoff,  each 
with  a  specified  probability.  When  the  decision  maker  is  risk  averse,  it  is  less  than  the 
expected  monetary  value.  (Clemen  and  Reilly,  2001:  535)  Also  called  CASH 
EQUIVALENT  by  Pratt  (1964b:  122)  and  CERTAIN  EQUIVALENT  by  Skinner  (1999: 
57).  See  RISK  PREMIUM. 
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CERTAINTY-EQUIVALENT  (CE)  ASSESSMENT  TECHNIQUE  -  Utility  function 
elicitation  employing  a  two-result  lottery  with  equal  probability  of  each  outcome  and  an 
alternative  with  a  certain  outcome  that  is  variable.  Adjustments  are  made  to  the  certain 
alternative  until  the  subject  is  ambivalent  between  the  uncertain  lottery  and  the  certain 
alternative.  (Clemen,  1996:  474-475)  This  method  is  referred  to  as  the  VARIABLE 
CERTAINTY  EQUIVALENT  METHOD  by  von  Winterfeldt  and  Edwards  (1986:  249). 

CLARITY  OF  ACTION  -  condition  of  the  decision  maker  knowing  the  appropriate 
action  although  the  outcome  is  uncertain.  (Spradlin,  1997). 

COEFFICIENT  OF  VALUE  SATIATION  -  See  VALUE  SATIATION. 

CONSISTENT  RISK  ATTITUDES  -  The  condition  of  the  decision  maker  maintaining 
risk  attitudes  on  all  evaluation  measures  that  are  consistent  with  respect  to  being  risk 
averse,  risk  neutral,  or  risk  seeking.  If  this  condition  is  present,  then  the  decision  maker  is 
risk  averse  (neutral)  (seeking)  on  all  evaluation  measures  if  she  is  risk  averse  (neutral) 
(seeking)  on  any  single  evaluation  measure. 

CONSTANT  RISK  ATTITUDES  -  The  condition  of  the  decision  maker  maintaining  a 
risk  attitude  throughout  the  domain  of  an  evaluation  measure  with  respect  to  being  risk 
averse,  risk  neutral,  or  risk  seeking.  If  this  condition  is  present,  then  the  decision  maker  is 
risk  averse  (neutral)  (seeking)  over  the  entire  domain  of  that  evaluation  measure  if  he  is 
risk  averse  (neutral)  (seeking)  at  any  point  in  the  domain  for  that  evaluation  measure. 
(Bell  and  Raiffa,  1988a) 

CONSTANT  RELATIVE  RISK  ATTITUDES  -  The  condition  of  the  decision  maker 
maintaining  a  risk  attitude  throughout  the  value  domain  (v(jc))  of  an  evaluation  measure 

with  respect  to  being  risk  averse,  risk  neutral,  or  risk  seeking.  If  this  condition  is  present, 
then  the  decision  maker  is  risk  averse  (neutral)  (seeking)  over  the  entire  domain  of  that 
evaluation  measure  if  he  is  risk  averse  (neutral)  (seeking)  at  any  point  in  the  domain  for 
that  evaluation  measure.  (Bell  and  Raiffa,  1988a) 

DECISION  -  “A  conscious,  irrevocable  allocation  of  resources  with  the  purpose  of 
achieving  a  desired  objective”  (Skinner,  1999:  11). 

DECISION  ANALYSIS  -  1.  A  structured  approach  in  analyzing  how  alternatives  for  a 
decision  would  produce  a  desired  result  (Decision  Analysis  Society,  November  3,  2000). 
2.  Analysis  of  a  decision  problem  strictly  by  employing  a  value  or  utility  function 
approach  (Keeney,  1982:  813). 

DECISION  ANALYSIS  CYCLE  -  “A  systematic  approach  to  solving  problems: 
STRUCTURING  a  problem  to  capture  the  essentials,  EVALUATION  to  gain  insight  and 
AGREEMENT  with  the  world  to  make  something  happen”  (Skinner,  1999:  356). 
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DECISION  MAKER  -  “Person  or  team  with  the  responsibility  and  authority  to  allocate 
resources  and  implement  the  decision”  (Skinner,  1999:  356). 

DECISION  MAKING  UNDER  RISK  -  Pertains  to  choice  among  lotteries  that  assign 
objective  probabilities  to  outcomes  (Kami,  1990). 

DECISION  MAKING  UNDER  UNCERTAINTY  -  Pertains  to  choice  among  alternative 
acts  that  assign  outcomes  to  states  of  nature  whose  likelihood  of  being  realized  is  a  matter 
of  subjective  belief  (Kami,  1990). 

DECISION  NODE  -  “A  point  in  a  decision  tree  where  a  decision  must  be  made” 
(Skinner,  1999:  357). 

DECISION  POLICY  -  A  rule  for  the  selection  of  alternatives.  Decision  policy  designates 
which  alternative  to  take  at  each  and  every  decision  node  in  a  decision  tree.  Also  referred 
to  as  a  DECISION  STRATEGY  or  STRATEGY.  (Skinner,  1999:  357) 

DECISION  THEORY  -  “The  mathematical  theory  of  decision  making  under 
uncertainty”  (Skinner,  1996:  357). 

DECISION  TREE  -  A  sequential  graphical  representation  of  decisions  and  uncertain 
events  that  provide  all  possible  outcomes  to  a  decision  situation.  (Skinner,  1999:  357) 

DYNAMIC  CONSISTENCY  -  Maintenance  of  alternative  choices  when  a  subject  is 
presented  a  decision  with  the  decision  in  differing  orders.  For  example,  if  a  decision 
made  with  consideration  of  later  probabilistic  events,  a  decision  maker  is  said  to  be 
dynamically  consistent  if  she  would  maintain  the  same  decision  choice  if  the  probabilistic 
events  were  known,  i.e.,  occurring  before  the  decision.  (Biswas,  1997:  15) 

ECONOMIC  VALUE  -  A  value  measured  in  units  of  currency  (Spradlin,  1997). 

ELASTICITY  -  The  relative  change  in  the  response  variable  to  a  relative  change  in  a 
predictor  variable, 


where  y  is  the  response  variable  and  xt  an  independent  variable  (Joish  and  Armstrong, 
2000:  537). 

ELLSBERG  PARADOX  -  Given  an  opaque  container  in  which  are  30  marbles,  ten  of 
red  and  the  balance  being  either  blue  or  yellow,  and  then  offered  two  sequential  bets:  (1) 
Call  red  or  blue,  if  correct  win  $5,000;  and  (2)  Call  red  or  blue,  if  wrong,  collect  $5,000. 
Ellsberg  found  that  most  subjects  choose  red  for  both  bets.  This  violates  expected  utility 
theory  because  red  as  the  choice  in  bet  (1)  evinces 
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pr  •w(5000)+(l-/?r)-w(0)>  ph  ■  i<(5000)+ (l -pb)  w(0) 
where  the  subscripts  pr  and  ph  indicate  the  respective  probabilities  of  selecting  a  marble 
of  that  color.  The  expected  utility  relationship  of  bet  (2)  is  captured  by 

pr  -w(o)  +  (l  — )-m(5000)>  pb  -u(o)  +  (l  -  pb)  w(5000) 
which  may  be  manipulated  to  form 

pr-u{5000)+(l-pr)-u{0)<pb  -m(0)+(1- pb)-u{0). 

This  clearly  violates  the  equation  resultant  from  bet  (1),  and  so  violates  expected  utility 
theory.  Ellsberg  advanced  the  argument  that  decision  makers  are  likely  to  place  more 
confidence  in  objective  rather  than  subjective  probabilities.  The  red  probabilities  are 
objective  while  those  for  blue  are  subjective,  leading  a  decision  maker  so  inclined  to 
select  red.  (Ellsberg,  1961:  653  -  655) 

ENCODING  UNCERTAINTY  -  “The  process  by  which  individual  judgments  about  an 
uncertain  variable  are  characterized  by  a  probability  distribution”  (Skinner,  1999:  357). 

EVALUATION  CONSIDERATION  -  (1)  “Any  matter  that  is  significant  enough  to  be 
taken  into  account  while  evaluating  alternatives.  Other  terms  sometimes  used  for 
evaluation  considerations  are  EVALUATION  CONCERN  or  AREA  OF  CONCERN.” 
(Kirkwood,  1997:  1 1-12)  (2)  “A  factor  to  compare  alternatives  (e.g.,  annual  income).” 
(Kirkwood,  1997,  as  quoted  by  Parnell,  undated) 

EVALUATION  MEASURE  -  “Measuring  scale  for  the  degree  of  attainment  of  an 
objective,”  e.g.,  annual  salary  in  dollars.  Also  called  measure  of  effectiveness,  measure  of 
merit,  or  metric.  (Kirkwood,  1997:  12) 

EXPECTED  MONETARY  VALUE  -  The  weighted  mean  of  outcomes,  expressed  in 
currency  units.  Weighting  is  provided  by  multiplying  the  outcome  by  the  corresponding 
probability.  (Skinner,  1999:  357). 

EXPECTED  UTILITY  -  The  theory  as  developed  by  von  Neumann  and  Morgenstern 
(1947)  that  the  alternative  may  be  rank  ordered  by  their  expected  utility.  The  expected 
utility  is  determined  by  multiplying  possible  outcomes  by  their  probabilities  and 
summing.  (Keller,  1992:  4)  Also  referred  to  as  UTILITY  THEORY^ 

EXPECTED  VALUE  -  1 .  “The  mean  of  the  values  one  would  expect  to  obtain  upon 
sampling  from  the  distribution  a  large  number  of  times”  (Decision  Analysis  Society, 
November  3,  2000).  Also  known  by  the  older  term  of  MATHEMATIC AL 
EXPECTATION.  2.  A  variation  of  EXPECTED  UTILITY  where  value  functions 
replace  utility  functions. 

FOURFOLD  PATTERN  OF  RISK  ATTITUDES  -  Tversky  and  Kahneman  (1992:  306  - 
307)  found  that  subject’s  risk  behavior  followed  a  pattern  where  risk  seeking  behavior 
was  demonstrated  for  low  probability  potential  gains  and  high  probability  potential 
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losses.  Risk  averse  behavior  was  presented  in  the  converse  cases.  This  is  summarized  in 
Table  44.  They  state  (1995:  1256)  that  numerous  studies  confirm  this  construct. 


Table  44.  Certainty  Equivalents  Under  Extremes  of  Probability  and  Prospects.  The 
Uncertain  Alternative  was  for  a  Gain  of  100  or  a  Loss  of  100,  with  Probability  of  Either 
0.05  or  0.95.  (Tversky  and  Kahneman,  1992:  307) 


Gain 

Loss 

Low  Probability 

w(l4)  =  0.05  ,  risk  seeking 

u(-  8)  =  -0.05 ,  risk  averse 

High  Probability 

w(95)  =  0.78  ,  risk  averse 

u{—  84)  =  -0.95 ,  risk  seeking 

FRAMING  -  The  manner  of  presentation  of  questions  employed  in  elicitation  in  decision 
analysis.  Framing  has  been  shown  to  be  a  significant  factor  affecting  the  answers 
received  from  the  decision  maker.  (Tversky  and  Kahneman,  1986) 

GOAL  -  (1)  “Threshold  of  achievement  with  respect  to  an  evaluation  consideration,” 
(Kirkwood,  1997:  12).  (2)  “Specific  degree  of  satisfaction  of  a  given  objective” 

(Decision  Analysis  Society,  November  3,  2000). 

INDEPENDENCE  AXIOM  -  “Consider  three  lotteries,  Lx ,  Z2 ,  and  Z3 .  Suppose  that  the 
relation  of  the  indifference  (or  strict  preference)  holds  between  Z,  and  L2 .  The  relation  of 
indifference  (or  strict  preference)  must  hold  between  the  compound  lotteries 
\LX  {a),  Z3  (l  -  a)]  and  [Z2  (a),  Z3  (l  -  a)]  ”  where  a  is  the  associated  probability  of  lotteries 

Zj  and  Z2 .  (Biswas,  1997:  1 1)  See  BETWEENESS  AXIOM. 

INDIFFERENCE  -  Indifference  between  two  objects  indicates  that  the  decision  maker 
desires  each  equally.  The  objects  are  considered  to  have  equal  value  or  utility.  (Skinner, 
1999:  357) 

INDIFFERENCE  CURVE  -  The  plot  of  continuous  points  in  n  attributes  where  the 
utility  (or  value)  is  constant  (Clemen,  1996:  540  -  541).  Also  known  as  isopreference 
curves.  The  concept  of  indifference  curves  was  conceived  by  the  economist  Edgeworth 
(Stigler,  1950:  325). 

INDIRECT  VALUES  -  “Things  that  the  decision  maker  values  that  are  not  likely  to 
show  up  on  accounting  statements.”  (Spradlin,  1997). 

INFLUENCE  DIAGRAM  -  A  graphical  representation  of  a  decision  situation.  Decisions 
are  represented  by  rectangles,  ovals  represent  stochastic  events,  diamonds  represent  the 
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final  outcome,  and  arcs  indicate  infonnation  status  and  conditional  relationships. 

(Clemen  and  Reilly,  2001:  52). 

LAYER  -  The  levels  in  the  value  tree.  Also  called  a  tier.  (Kirkwood,  1997b,  as  quoted 
by  Parnell,  undated) 

LEVEL  -  “Specific  numerical  rating  for  a  particular  alternative  with  respect  to  a 
specified  evaluation  measure,”  e.g.,  $55K;  also  called  a  SCORE  (Kirkwood,  1997b:  12, 
55). 

LOCAL  RISK  AVERSION  -  Pratt  (1964)  introduced  a  measure  of  local  risk  aversion, 
r(x)  =  - u"(x)/u(x ) .  This  was  also  independently  developed  by  Arrow. 

MEASURABLE  VALUE  FUNCTION  -  “A  value  function  that  may  be  used  to  order  the 
differences  in  the  strength  of  preference  between  pairs  of  alternatives”  (Dyer  and  Sarin, 
1979:810). 

OBJECTIVE  -  (1)  “The  desired  level  of  performance  against  a  value”  (Skinner,  1999: 
358).  (2)  “Preferred  direction  of  attainment  with  respect  to  an  evaluation  consideration” 
(e.g.,  higher  annual  income).  See  STRATEGIC  OBJECTIVE.  (Kirkwood,  1997:  12)  (3) 
“What  the  decision  maker  hopes  to  achieve  by  allocating  the  resources”  (Spradlin,  1997). 

OPTION  -  “An  alternative  that  permits  a  future  decision  following  revelation  of 
information.  All  options  are  alternatives,  but  not  all  alternatives  are  options.”  (Spradlin, 
1997). 

ORDINAL  UTILITY  FUNCTION  -  See  VALUE  FUNCTION. 

OUTCOME  -  “The  result  of  the  decision  situation  measured  on  the  scale  of  the  decision 
maker’s  values”  (Spradlin,  1997). 

OUTCOME  BRANCH  -  “A  branch  emanating  from  a  probability  node  representing  the 
possible  outcome  and  its  probability  of  occurrence”  (Skinner,  1999:  358). 

PORTFOLIO  PROBLEM  -  “The  various  decisions  faced  in  the  strategy  are  of  a  similar 
nature,  and  the  decision  maker  does  not  have  sufficient  resources  for  funding  all 
combinations  of  alternatives.”  (Spradlin,  1997). 

PREFERENCE  -  “The  decision  maker’s  attitude  toward  the  value,  timing,  and 
uncertainty  of  outcomes”  (Skinner,  1999:  358). 

PREFERENCE  AREA  -  (1)  The  area  under  the  utility  curve,  plotted  in  a  unit  square, 
that  serves  as  a  measure  of  the  decision  maker’s  risk  attitude.  Areas  below  0.45  indicate 
risk  aversion,  above  0.55  indicate  risk  seeking,  and  between  these  values  indicates  risk 
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neutrality.  (Kimbrough  and  Weber,  1994:  627)  (2)  More  generally,  the  same  definition 
applied  to  any  preference  function. 

PREFERENCE  FUNCTION  -  Either  a  value  or  a  utility  function. 

PROBABILITY  -  “A  number  between  zero  and  one  (inclusively)  representing  the  degree 
of  belief  a  person  attaches  to  the  occurrence  of  an  event”  (Skinner,  1999:  358). 

PROBABILITY-EQUIVALENT  (PE)  ASSESSMENT  TECHNIQUE  -  Utility  function 
elicitation  employing  a  two-result  lottery  where  the  probability  p  of  the  desired  outcome 
is  adjusted,  the  undesired  outcome  occurs  with  probability  (1-p),  and  the  alternative  with 
the  certain  outcome  is  fixed.  Adjustments  are  made  to  p  until  the  subject  is  ambivalent 
between  the  uncertain  lottery  and  the  certain  alternative.  (Clemen,  1996:  475)  This 
method  is  called  the  variable  probability  method  by  von  Winterfeldt  and  Edwards  (von 
Winterfeldt,  1986:  243),  who  also  say  the  method  is  referred  to  as  the  basic  reference 
lottery  ticket  (BRLT)  method. 

PROBABILITY  NODE  -  “A  point  in  a  decision  tree  where  an  uncertainty  will  be 
resolved;  often  called  chance  node”  (Skinner,  1999:  358). 

RATING  -  v{x)  where  v  is  a  value  function;  also  called  a  “value.”  (Kirkwood,  1997b: 

55). 

RESPONSE  SURFACE  METHODOLOGY  (RSM)  -  A  collection  of  knowledge  and 
techniques  from  statistical  experimental  design,  regression  modeling,  and  optimization 
(Myers  and  Montgomery,  1995:  xiii). 

RISK  -  (1)  the  chance  of  loss;  (2)  the  possibility  of  loss;  (3)  uncertainty;  (4)  the 
dispersion  of  actual  from  expected  results;  (5)  the  probability  of  any  outcome  different 
from  the  one  expected;  (6)  the  probability  of  an  event  times  the  probable  cost  (or  loss)  if 
the  event  occurs  (Dargahi-Noubary  &  Growney,  1998);  (7)  the  condition  under  which 
alternatives  involve  known  probabilities. 

RISK  AVERSION  -  See  LOCAL  RISK  AVERSION. 

RISK  AVERSION  COEFFICIENT  -  When  an  exponential  utility  function  is  employed 
“  u  (x)  =  a  —  be  /R  where  x  is  the  value  (such  as  dollars),  R  is  the  risk  tolerance  and  a  and 
b  are  parameters  set  by  the  choice  of  two  points  in  the  utility  curve. .  .One  sometimes  sees 
reference  to  the  RISK  AVERSION  COEFFICIENT,  which  is  defined  as  j/^  ” 

(McNamee,  1987:  94).  See  also  LOCAL  RISK  AVERSION  and  PREFERENCE  AREA. 

RISK  PREMIUM  -  The  expected  monetary  value  of  an  uncertain  alternative  less  the 
CERTAINTY  EQUIVALENT  of  the  decision  maker.  It  is  a  measure  of  the  decision 
maker’s  risk  aversion.  (Clemen  and  Reilly,  2001:  534) 
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RISK  TOLERANCE  -  “Describes  the  decision  maker’s  attitude  toward  risk”  (Spradlin, 
1997). 

RISK- AVERSE  DECISION  MAKER  -  “One  who  values  alternatives  at  less  than  their 
expected  values”  (Spradlin,  1997). 

RISK-NEUTRAL  DECISION  MAKER  -  “One  who  is  willing  to  play  the  long-run  odds 
when  making  decisions  and  will  evaluate  alternatives  according  to  their  expected  values” 
(Spradlin,  1997). 

RISK-SEEKING  DECISION  MAKER  -  One  who  values  an  alternative  more  than  the 
expected  value. 

REQUISITE  DECISION  MODEL  -  “A  model  whose  form  and  content  are  sufficient  to 
solve  a  particular  problem.”  (Phillips,  1984:  29) 

SCORE  -  See  LEVEL, 

SCORING  FUNCTION  -  “A  single  dimensional  value  function  that  assigns  value  to  an 
evaluation  measure  level”  (Kirkwood,  1997b,  as  quoted  by  Parnell,  undated). 

SENSITIVITY  ANALYSIS  -  (1)  “Examining  the  impact  of  reasonable  changes  in  base- 
case  assumptions  (Eschenbach,  1992:  41).  (2)  “Measuring  the  impact  of  each  uncertainty 
on  the  value  of  an  alternative  to  detennine  which  are  critical,  i.e.,  would  change  the  order 
of  preference  for  the  alternatives”  (Skinner,  1999:  358). 

SENSITIVITY  CHART  -  See  TORNADO  DIAGRAM  (Skinner,  1999:  358). 

SIMPLE  DECISION  -  “One  in  which  there  is  only  one  decision  to  be  made,  even  though 
there  might  be  many  alternatives.”  (Spradlin,  1997). 

SPIDERPLOT  -  A  Cartesian  plot  of  response  the  dependent  variable  to  percentage 
change  in  independent  variables  where  each  independent  variable  is  represented  by  a 

curve.  The  curves  intersect  at  the  case  point  (100%,E'[m(jc)])  creating  a  spider  web-like 

appearance.  The  plot  shows  sensitivity  of  the  decision  problem. 

STRATEGIC  EQUIVALENCE  -  (1)  Condition  under  which  preference  functions 
provide  identical  rank  orderings  of  a  set  of  alternatives  (Kirkwood,  1997b:  229  and  245). 
(2)  Condition  under  which  methodologies  produce  identical  rank  orderings  of 
alternatives. 

STRATEGIC  OBJECTIVE  -  “The  ultimate  objective  for  the  decision”  (Kirkwood, 

1997b,  as  quoted  by  Parnell,  undated). 
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STRATEGY  -  A  set  of  alternatives  for  sequential  decisions  in  a  decision  situation  where 
the  alternatives  follow  some  policy  or  principle.  For  example,  one  strategy  may  involve 
only  conservative  alternatives  while  another  competing  strategy  employs  aggressive 
choices.  Also  known  s  a  POLICY.  (Skinner,  1999:  358). 

STRATEGY  GENERATION  TABLE  -  Employed  to  create  or  display  strategies.  The 
first  column  contains  titles  for  generated  strategies.  Decisions  are  assigned  to  each 
column.  The  alternatives  populate  the  column.  A  strategy  is  indicated  by  linking  the 
name  to  each  alternative  chosen  under  that  strategy.  Strategy  names  are  chosen  to 
highlight  differences  among  the  strategies.  (Kirkwood,  1997b:  47  -  48) 

STRATEGY  TABLE  -  See  Strategy  Generation  Table  (Skinner,  1999:  358). 

SUBJECTIVE  EXPECTED  UTILITY  -  The  EXPECTED  UTILITY  approach  where 
objective  probabilities  are  replaced  with  subjective  probabilities.  The  idea  was 
introduced  by  Ramsey  (1954).  (Keller,  1992:  4) 

SUBJECTIVE  PROBABILITY  -  “Numerical  specification  of  an  individual’s  degree  of 
belief  that  an  uncertain  event  will  occur  (Kirkwood,  1997:  109). 

TORNADO  DIAGRAM  -  A  graphical  representation  of  the  decision  analysis  model 
optimal  solution  response  to  variation  in  model  parameters.  The  diagram  is  a  series  of 
horizontal  bars,  each  representing  the  response  variation.  By  convention  the  bars  are 
ordered  in  descending  impact,  creating  an  inverted  triangle  reminiscent  of  an  atmospheric 
tornado.  Also  known  as  a  SENSITIVITY  CHART. 

TRADE-OFFS  -  “Judgments  about  how  much  a  decision  maker  is  willing  to  sacrifice  on 
one  value  in  order  to  receive  more  of  another”  (Decision  Analysis  Society,  November  3, 
2000). 

UNCERTAINTY  -  (1)  “Any  event  for  which  the  outcome  is  not  known  at  the  time  a 
decision  is  made”  (Skinner,  1999:  359).  (2)  “Uncertainty  means  unknown  probabilities,” 
(Bernstein,  1996:  133).  This  suggests  a  distinction  beyond  the  division  between 
deterministic  and  stochastic,  or  that  a  stochastic  situation  may  either  be  known,  and  still 
certain,  and  unknown  (uncertain).  (3)  “Uncontrollable  elements”  (Spradlin,  1997). 

UTILITY  -  “A  mathematical  function  (curve)  that  represents  the  decision  maker’s  risk 
attitude”  (Skinner,  1999:  359). 

UTILITY  FUNCTION  —  (1)  “. . .  a  real-valued  function  defined  on  a  set  and  inducing  or 
corresponding  to  a  preference  order  by  having  u(x)<=u(y)  exactly  when  y  is  preferred  to 
x”  (Borowski,  1991:  620).  [Note  that  this  is  a  definition  from  a  non-DA  source.  The  term 
VALUE  FUNCTION  does  not  appear  in  this  source.]  (2)  “...  encodes  a  decision  maker’s 
attitude  toward  risk  taking  in  mathematical  form  by  relating  the  decision  maker’s 
satisfaction  with  the  outcome  (or  “utility”  associated  with  the  outcome)  to  the  monetary 
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value  of  the  outcome  itself.”  (Decision  Analysis  Society,  November  3,  2000).  (3)  “A 
preference  function  under  uncertainty”  (Dyer  and  Sarin,  1979:  810). 

UTILITY  THEORY  -  See  EXPECTED  UTILITY. 

TIER  -  See  LAYER. 

VALUE  -  (1)  “An  outcome  measure,  e.g.,  NPV”  (Skinner,  1996).  (2)  “  v(x)  where  v  is  a 
value  function;  also  called  a  ‘rating’”  (Kirkwood,  1997b:  55). 

VALUE  FUNCTION  -  (1)  A  function  v(x)  is  a  value  function  if  it  is  true  that 
v(x')  >  v(x")  i  ff  x  >-  x"  where  x'  and  x"  arc  specified  but  arbitrary  levels  of  x  (Kirkwood, 
1997:  229).  (2)  Used  under  conditions  of  certainty  to  rank-order  alternatives;  also  called 
ORDINAL  UTILITY  FUNCTIONS  (Clemen,  1996:  552).  (3)  “A  preference  function 
under  certainty”  (Dyer  and  Sarin,  1979:  810). 

VALUE  HIERARCHY  -  (1)  “A  value  structure  with  a  hierarchical  structure”;  also  called 
a  VALUE  TREE.  (Kirkwood,  1997:  12).  (2)  “Pictorial  representation  of  the  structure  of 
the  evaluation  considerations”  (Kirkwood,  1997b,  as  quoted  by  Parnell,  undated). 

VALUE  OF  INFORMATION  -  “The  maximum  price  one  should  pay  for  knowing  the 
actual  value  of  uncertainty  prior  to  making  the  decision”  (Skinner,  1999:  359). 

VALUE  MODEL  -  “A  mathematical  model  of  the  value  structure  that  includes  scoring 
functions  and  weights”  (Kirkwood,  1997b,  as  quoted  by  Parnell,  undated). 

VALUE  SATIATION  -  A  construct  introduced  by  Dyer  and  Sarin  (1982:  877)  for 
measurable  value  functions  that  is  the  analog  of  Pratt’s  local  risk  aversion  for  utility 
functions.  The  coefficient  of  value  satiation,  defined  m(x)  =  - v"{x)jv\x ) ,  is  a  local 
measure  of  strength  of  preference  at  asset  level  x. 

VALUE  STRUCTURE  -  “The  entire  set  of  evaluation  considerations,  objectives,  and 
evaluation  measures  for  a  particular  decision  analysis.”  (Kirkwood,  1997b:  12). 

VALUE  TREE  -  See  VALUE  HIERARCHY. 

VALUES  -  Things  that  are  important  to  you  in  a  general  sense  (Clemen,  1996:  19). 

VARIABLE  CERTAINTY  EQUIVALENT  METHOD  -  See  CERTAINTY- 
EQUIVALENT  ASSESSMENT  TECHNIQUE. 

VARIABLE  PROBABILITY  METHOD  -  See  PROBABILITY-EQUIVALENT 
ASSESSMENT  TECHNIQUE. 
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WEAK  INDEPENDENCE  AXIOM  -  “Consider  three  lotteries,  Lx ,  L2,  and  A  and  an 
individual  who  is  indifferent  between  Lx  and  L2 .  Given  an  y  compound  lottery 
\LX  (p),L2  (l  -  p)]  there  must  exist  a  q  such  that  the  individual  is  indifferent  between  this 
compound  lottery  and  \L2(q),L3(\- q)],”  where  p  and  q  are  the  associated  probabilities 
of  lotteries  Lx  and  L2 .  Normally  this  axiom  is  employed  in  conjunction  with  the 
BETWEENESS  AXIOM  to  guarantee  non-intersection  of  indifference  curves.  (Biswas, 
1997:  13) 

WEIGHTS  -  “Our  relative  preference  for  evaluation  considerations  and  evaluation 
measures.  Weights  depend  on  the  domain  of  the  scoring  functions.”  (Kirkwood,  1997b, 
as  quoted  by  Parnell,  undated) 
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Appendix  B.  Tactical  Questionnaire  Detailed  Results 


Subject  FJ1 


Artillery 


Potential  Ambush  Sites 


-U(CE) 
■U (PE) 


125  250  375  500 

Miles  of  Exposure 


Fuel  Remaining 


-U(CE) 
"U (PE) 


75  100 


Percent  Remaining 


Figure  93.  Subject  FJ 1  ’s  Preference  Functions. 


Subject  FJ2 


Figure  94.  Subject  FJ2’s  Preference  Functions. 


Subject  FJ3 


Figure  95.  Subject  FJ3’s  Preference  Functions. 


Subject  FJ4 


Figure  96.  Subject  FJ4’s  Preference  Functions. 


GO] 


Subject  FJ5 


Figure  97.  Subject  FJ5’s  Preference  Functions. 
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Subject  FJ6 


Figure  98.  Subject  FJ6’s  Preference  Functions. 


Subject  FJ7 


Figure  99.  Subject  FJ7’s  Preference  Functions. 


Subject  FJ8 


Subject  FJ9 


Travel  Time 


- Value 

-■-U  (CE) 
^  U  (PE) 


Figure  101.  Subject  FJ9’s  Preference  Functions. 
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Subject  FJ10 


Fuel  Remaining 


-U(CE) 
'  U  (PE) 


75  100 


Percent  Remaining 


Travel  Time 


Figure  102.  Subject  FJlO’s  Preference  Functions. 
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Subject  FJ1 1 


Multiattribute  Utility 

1.00 
0.90 
0.80 
0.70 
0.60 
0.50 
0.40 
0.30 
0.20 
0.10 
0.00 
0 


.00  0.50  1.00 

Value 


Figure  103.  Subject  FJ1  l’s  Preference  Functions. 
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Subject  FJ12 


Figure  104.  Subject  FJ12’s  Preference  Functions. 


Subject  F13 
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utility(upper)  — b — utility(lower) 


Figure  105.  Subject  FJ13’s  Preference  Functions. 
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Subject  FJ14 
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Figure  106.  Subject  FJ14’s  Preference  Functions. 
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Subject  FJ15 


Figure  107.  Subject  FJ1 5 ’s  Preference  Functions. 


Subject  FJ16 


Figure  108.  Subject  FJ16’s  Preference  Functions. 


Multiattribute  Utility 


Subject  FJ17 


Figure  109.  Subject  FJ17’s  Preference  Functions. 


Subject  FJ18 


Figure  110.  Subject  FJ1 8 ’s  Preference  Functions. 


254 


Subject  FJ19 


Figure  111.  Subject  FJ19’s  Preference  Functions. 
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Subject  FJ20 


Figure  112.  Subject  FJ20’s  Preference  Functions. 


Table  45.  Subjects’  Attribute  Weightings  and  Utility  Data. 


ID 

Weights 

CE  Utility 

PE  Utility 

Multattribute 

Utility 

Att 

W; 

Att 

0.25 

0.5 

0.75 

Att 

Qi 

q2 

q3 

Att 

X; 

FJ1 

FA 

0.27 

FA 

325 

275 

100 

FA 

0.30 

0.55 

0.80 

FA 

Amb 

0.40 

Amb 

40 

25 

12 

Amb 

0.30 

0.55 

0.80 

Amb 

Fuel 

0.22 

Fuel 

27 

57 

65 

Fuel 

0.80 

0.55 

0.30 

Fuel 

Time 

0.11 

Time 

20 

18.5 

14 

Time 

0.30 

0.55 

0.80 

Time 

FJ2 

FA 

0.17 

FA 

300 

235 

170 

FA 

0.03 

0.25 

0.63 

FA 

0 

Amb 

0.33 

Amb 

37 

32 

22 

Amb 

0.23 

0.90 

0.98 

Amb 

0 

Fuel 

0.33 

Fuel 

5 

50 

98 

Fuel 

0.88 

0.50 

0.13 

Fuel 

100 

Time 

0.17 

Time 

17 

14 

13 

Time 

0.13 

0.50 

0.70 

Time 

24 

FJ3 

FA 

0.45 

FA 

325 

200 

90 

FA 

0.35 

0.70 

wnm 

FA 

0 

Amb 

0.30 

Amb 

27 

22 

7 

Amb 

0.23 

0.35 

wxm 

Amb 

50 

Fuel 

0.21 

Fuel 

12 

38 

60 

Fuel 

0.76 

0.65 

wzm 

Fuel 

100 

Time 

0.04 

Time 

24 

18 

16 

Time 

0.62 

0.75 

mm 

Time 

24 

FJ4 

FA 

0.25 

FA 

265 

250 

125 

FA 

0.18 

0.45 

0.67 

FA 

Amb 

0.31 

Amb 

20 

Amb 

0.20 

0.40 

0.76 

Amb 

Fuel 

0.27 

Fuel 

60 

Fuel 

0.85 

0.50 

0.20 

Fuel 

Time 

0.18 

Time 

17 

Time 

0.20 

0.50 

0.75 

Time 

FJ5 

FA 

0.05 

FA 

75 

FA 

0.15 

0.50 

0.75 

FA 

0 

Amb 

0.45 

Amb 

20 

Amb 

0.36 

0.60 

0.84 

Amb 

50 

Fuel 

0,17 

Fuel 

55 

Fuel 

0.80 

0.50 

0.20 

Fuel 

0 

Time 

0.33 

Time 

16 

Time 

0.51 

0.60 

0.80 

Time 

24 

FJ6 

FA 

0.22 

FA 

360 

315 

150 

FA 

0.16 

0.40 

0.64 

FA 

Amb 

0.44 

Amb 

40 

30 

15 

Amb 

0.16 

0.40 

0.64 

Amb 

Fuel 

0.11 

Fuel 

15 

40 

60 

Fuel 

0.64 

0.40 

0.16 

Fuel 

Time 

0.22 

Time 

19 

16 

14 

Time 

0.16 

0.40 

0.64 

Time 

FJ7 

FA 

0.17 

FA 

350 

250 

75 

FA 

0.16 

0.40 

0.64 

FA 

500 

Amb 

0.33 

Amb 

40 

30 

15 

Amb 

0.26 

0.70 

0.76 

Amb 

0 

Fuel 

0.17 

Fuel 

40 

65 

80 

Fuel 

0.52 

0.40 

0.12 

Fuel 

100 

Time 

0.33 

Time 

20 

18 

14 

Time 

0.28 

0.40 

0.64 

Time 

24 

FJ8 

FA 

0.63 

FA 

310 

275 

75 

FA 

0.20 

0.50 

0.75 

FA 

500 

Amb 

0.13 

Amb 

26 

20 

9 

Amb 

0.35 

0.50 

0.85 

Amb 

0 

Fuel 

0.20 

Fuel 

25 

50 

75 

Fuel 

0.75 

0.70 

0.35 

Fuel 

100 

Time 

0.05 

Time 

21 

18 

13.5 

Time 

0.25 

0.70 

0.73 

Time 

12 

FJ9 

FA 

0.42 

FA 

425 

300 

200 

FA 

0.16 

0.40 

0.70 

FA 

Amb 

0.21 

Amb 

37 

35 

25 

Amb 

0.20 

0.50 

0.70 

Amb 

Fuel 

0.12 

Fuel 

55 

65 

70 

Fuel 

0.76 

0.40 

0.24 

Fuel 

Time 

0.24 

Time 

16 

13 

13 

Time 

0.08 

0.40 

0.52 

Time 

FJ10 

FA 

0.25 

FA 

300 

200 

60 

FA 

0.25 

0.50 

0.80 

FA 

Amb 

0.50 

Amb 

32 

25 

12 

Amb 

0.25 

0.50 

0.75 

Amb 

Fuel 

0.17 

Fuel 

20 

40 

65 

Fuel 

0.75 

0.50 

0.25 

Fuel 

Time 

0.08 

Time 

20 

18 

16 

Time 

0.30 

0.50 

0.75 

Time 
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Table  45.  Subjects’  Attribute  Weightings  and  Utility  Data,  continued. 


ID 

Weights 

CE  Utility 

PE  Utility 

Multattribute 

Utility 

Att 

W; 

Att 

0.25 

0.5 

0.75 

Att 

Qi 

q2 

q3 

Att 

X; 

FJ1 1 

FA 

0.53 

FA 

325 

250 

75 

FA 

0.24 

0.40 

0.46 

FA 

0 

Amb 

0.27 

Amb 

30 

Amb 

0.24 

0.40 

0.76 

Amb 

0 

Fuel 

0.12 

Fuel 

25 

Fuel 

0.90 

0.80 

0.64 

Fuel 

0 

Time 

0.08 

Time 

20 

Time 

0.35 

0.50 

0.90 

Time 

12 

FJ12 

FA 

0.55 

FA 

300 

300 

175 

FA 

0.25 

0.50 

0.75 

FA 

Amb 

0.11 

Amb 

35 

25 

15 

Amb 

0.36 

0.60 

0.80 

Amb 

Fuel 

0.30 

Fuel 

45 

70 

80 

Fuel 

0.88 

0.60 

0.36 

Fuel 

Time 

0.03 

Time 

12.15 

12.1 

12.05 

Time 

1 

1 

1 

Time 

FJ13 

FA 

0.61 

FA 

350 

250 

100 

FA 

0.30 

0.60 

0.82 

FA 

Amb 

0.06 

Amb 

35 

20 

11 

Amb 

0.30 

0.50 

0.80 

Amb 

Fuel 

0.17 

Fuel 

53 

70 

72 

Fuel 

0.78 

0.50 

0.25 

Fuel 

Time 

0.17 

Time 

22 

17 

14.5 

Time 

0.18 

0.50 

0.63 

Time 

FJ14 

FA 

0.45 

FA 

300 

200 

125 

FA 

0.16 

0.40 

0.64 

FA 

0 

Amb 

0.38 

Amb 

42 

30 

15 

Amb 

0.20 

0.50 

0.70 

Amb 

50 

Fuel 

0.08 

Fuel 

35 

50 

67.5 

Fuel 

0.64 

0.40 

0.20 

Fuel 

0 

Time 

0.08 

Time 

22 

19 

17 

Time 

0.16 

0.40 

0.64 

Time 

12 

FJ15 

FA 

0.42 

FA 

400 

250 

175 

FA 

0.25 

0.50 

0.75 

FA 

0 

Amb 

0.42 

Amb 

39 

35 

17 

Amb 

0.25 

0.50 

0.75 

Amb 

0 

Fuel 

0.11 

Fuel 

10 

25 

35 

Fuel 

0.93 

0.50 

0.35 

Fuel 

0 

Time 

0.06 

Time 

22 

20 

17 

Time 

0.63 

0.90 

0.99 

Time 

24 

FJ16 

FA 

0.10 

FA 

250 

200 

100 

FA 

0.10 

0.50 

0.75 

FA 

Amb 

0.81 

Amb 

27 

25 

12 

Amb 

0.12 

0.60 

0.76 

Amb 

Fuel 

0.07 

Fuel 

50 

70 

75 

Fuel 

0.70 

0.50 

0.25 

Fuel 

Time 

0.02 

Time 

22 

20 

15 

Time 

0.36 

0.60 

0.68 

Time 

FJ17 

FA 

0.19 

FA 

340 

300 

150 

FA 

0.36 

0.55 

0.82 

FA 

0 

Amb 

0.56 

Amb 

35 

30 

15 

Amb 

0.33 

0.60 

0.82 

Amb 

50 

Fuel 

0.17 

Fuel 

25 

50 

65 

Fuel 

0.84 

0.35 

0.12 

Fuel 

100 

Time 

0.08 

Time 

23 

21 

13 

Time 

0.64 

0.75 

0.91 

Time 

24 

FJ18 

FA 

0.08 

FA 

235 

220 

150 

FA 

0.30 

0.60 

0.80 

FA 

Amb 

0.08 

Amb 

0 

0 

0 

Amb 

0.12 

0.30 

0.51 

Amb 

Fuel 

0.21 

Fuel 

3 

10 

30 

Fuel 

0.80 

0.60 

0.42 

Fuel 

Time 

0.62 

Time 

23.5 

16 

14 

Time 

0.36 

0.60 

0.76 

Time 

FJ19 

FA 

0.38 

FA 

325 

200 

75 

FA 

0.33 

0.65 

0.81 

FA 

500 

Amb 

0.38 

Amb 

30 

30 

16 

Amb 

0.28 

0.50 

0.73 

Amb 

50 

Fuel 

0.17 

Fuel 

10 

25 

45 

Fuel 

0.99 

0.85 

0.64 

Fuel 

100 

Time 

0.08 

Time 

22 

18 

16 

Time 

0.20 

0.65 

0.91 

Time 

12 

FJ20 

FA 

0.17 

FA 

350 

200 

125 

FA 

0.25 

0.50 

0.80 

FA 

0 

Amb 

0.33 

Amb 

42 

35 

20 

Amb 

0.36 

0.60 

0.76 

Amb 

0 

Fuel 

0.33 

Fuel 

30 

50 

75 

Fuel 

0.72 

0.60 

0.24 

Fuel 

0 

Time 

0.17 

Time 

21 

18 

15 

Time 

0.42 

0.70 

0..88 

Time 

12 
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Table  46.  Subjects’  Value  Function  Data. 


ID 

FA 

Ambush 

Fuel 

Time 

FJ1 

X 

0 

125 

375 

500 

0 

10 

20 

40 

0 

33 

100 

0 

21 

24 

V 

1 

.8 

.4 

0 

1 

.8 

.7 

0 

0 

0 

1 

1 

0 

0 

FJ2 

X 

0 

50 

300 

500 

1 

2 

3 

0 

66 

75 

100 

12 

18 

21 

24 

V 

1 

1 

0 

0 

.5 

.4 

.3 

0 

0 

.8 

1 

1 

.8 

0 

0 

FJ3 

X 

0 

125 

375 

500 

0 

10 

20 

35 

0 

25 

75 

100 

12 

15 

18 

24 

V 

1 

.9 

.2 

0 

1 

.8 

.15 

0 

0 

0 

.8 

1 

1 

1 

.15 

0 

FJ4 

X 

0 

125 

250 

500 

0 

23 

50 

0 

33 

100 

12 

21 

24 

V 

1 

.4 

.1 

0 

1 

.1 

0 

0 

0 

1 

1 

0 

0 

FJ5 

X 

0 

375 

500 

0 

25 

50 

0 

40 

100 

12 

20 

24 

V 

1 

0 

0 

1 

0 

0 

0 

0 

1 

1 

0 

0 

FJ6 

X 

0 

300 

500 

0 

10 

50 

0 

25 

35 

50 

12 

15 

18 

21 

V 

1 

0 

0 

1 

0 

0 

0 

.3 

.3 

.5 

1 

.9 

.5 

.25 

X 

60 

70 

100 

24 

V 

.77 

.9 

1 

0 

FJ7 

X 

50 

125 

250 

400 

0 

35 

50 

25 

50 

75 

100 

12 

15 

21 

24 

V 

.9 

.9 

.5 

0 

1 

0 

0 

.25 

.75 

1 

1 

1 

.9 

0 

0 

FJ8 

m 

250 

375 

500 

0 

10 

40 

50 

25 

50 

75 

90 

15 

18 

22 

24 

D 

.4 

0 

0 

1 

.9 

.05 

0 

0 

.25 

.8 

.95 

1 

.75 

0 

0 

FJ9 

X 

0 

125 

375 

400 

0 

28 

50 

0 

15 

80 

100 

12 

20 

24 

V 

l 

.8 

.45 

0 

1 

0 

0 

0 

0 

1 

1 

1 

0 

0 

FJ10 

X 

0 

125 

250 

275 

0 

10 

20 

25 

0 

25 

50 

75 

12 

15 

18 

21 

V 

l 

.85 

.3 

0 

1 

.7 

.5 

.3 

0 

.2 

.4 

1 

1 

1 

.3 

.1 

X 

500 

30 

40 

50 

100 

24 

V 

l 

.2 

0 

0 

1 

0 

FJ1 1 

X 

0 

125 

300 

500 

0 

20 

30 

50 

0 

33 

100 

12 

16 

20 

24 

V 

l 

.75 

0 

0 

1 

.8 

0 

0 

0 

0 

1 

1 

.8 

0 

0 

FJ12 

X 

0 

125 

500 

0 

30 

50 

0 

50 

75 

100 

12 

21 

24 

V 

l 

.1 

0 

1 

.1 

0 

0 

0 

.8 

1 

1 

.1 

0 

FJ13 

H 

250 

350 

10 

20 

30 

40 

25 

50 

75 

100 

12 

15 

21 

24 

D 

.6 

0 

.9 

.75 

.4 

0 

.1 

.5 

.9 

1 

1 

.9 

.1 

0 

FJ14 

X 

0 

100 

300 

500 

0 

20 

50 

0 

25 

100 

12 

15 

18 

24 

V 

l 

1 

0 

0 

1 

0 

0 

0 

0 

1 

1 

.95 

.1 

0 

FJ15 

X 

0 

50 

250 

500 

10 

20 

25 

50 

0 

25 

50 

100 

12 

20 

24 

V 

1 

.3 

.1 

0 

.3 

.2 

.1 

0 

0 

0 

1 

1 

1 

.1 

0 

FJ16 

X 

50 

125 

250 

500 

0 

2 

10 

50 

65 

75 

90 

100 

12 

15 

22 

24 

V 

.9 

.5 

.1 

0 

1 

.75 

.2 

0 

.3 

.8 

1 

1 

1 

.8 

0 

0 

FJ17 

X 

0 

300 

500 

0 

35 

50 

0 

75 

100 

12 

21 

24 

V 

1 

.05 

0 

1 

.05 

0 

0 

.05 

1 

1 

0 

0 

FJ18 

X 

0 

375 

500 

0 

30 

50 

0 

25 

100 

12 

15 

21 

24 

V 

1 

.5 

0 

1 

.3 

0 

0 

.15 

1 

1 

.85 

.5 

0 

FJ19 

X 

0 

125 

500 

0 

3 

15 

50 

0 

50 

80 

100 

12 

18 

24 

V 

1 

.9 

0 

1 

1 

0 

0 

0 

.2 

.9 

1 

1 

.1 

0 

FJ20 

X 

0 

250 

500 

0 

25 

50 

0 

50 

75 

100 

12 

15 

18 

24 

V 

1 

.1 

0 

1 

.1 

0 

0 

.1 

.9 

1 

1 

.9 

.1 

0 
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Table  47.  Subjects’  Elicited  Simultaneous  Equation  Coefficients. 


ID 

FA 

Ambush 

Fuel 

Time 

FJ1 

100 

25 

50 

14 

250 

10 

75 

14 

200 

28 

12 

300 

8 

5 

24 

FJ4 

300 

20 

50 

18 

300 

35 

100 

12 

400 

15 

50 

19 

200 

12 

20 

24 

FJ6 

100 

4 

100 

18 

300 

2 

60 

300 

1 

40 

12 

125 

10 

60 

19 

FJ9 

100 

25 

70 

18 

400 

18 

50 

18 

75 

20 

30 

200 

10 

90 

FJ10 

450 

25 

18 

200 

22 

300 

15 

45 

24 

50 

28 

18 

FJ12 

300 

30 

80 

15 

100 

0 

40 

21 

50 

10 

50 

24 

300 

5 

50 

18 

FJ13 

425 

25 

60 

18 

350 

50 

30 

15 

75 

0 

30 

21 

200 

15 

50 

24 

FJ16 

75 

0 

55 

18 

75 

15 

75 

18 

0 

10 

55 

0 

5 

30 

FJ18 

10 

18 

10 

80 

12 

16 

60 

14 

15 

100 

18 
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Table  48.  Subjects’  Demographic  Data. 


ID 

Ver 

YOS 

CA/NCA 

Age 

Gender 

Grade 

FJ1 

B 

6 

CA 

29 

M 

03 

FJ2 

A 

26 

CA 

47 

M 

E9 

FJ3 

A 

2 

NCA 

24 

M 

02 

FJ4 

B 

14 

NCA 

33 

M 

03 

FJ5 

B 

14 

CA 

32 

M 

E6 

FJ6 

A 

24 

CA 

49 

M 

04 

FJ7 

B 

2 

NCA 

21 

M 

E2 

FJ8 

A 

10 

NCA 

44 

M 

03 

FJ9 

B 

16 

CA 

37 

M 

E7 

FJ10 

B 

5 

CA 

29 

M 

03 

FJ1 1 

B 

11 

CA 

31 

M 

03 

FJ12 

A 

17 

CA 

36 

M 

E8 

FJ13 

B 

6 

NCA 

27 

F 

03 

FJ14 

B 

3.5 

NCA 

25 

F 

02 

FJ15 

B 

8 

CA 

29 

M 

E6 

FJ16 

A 

9 

CA 

28 

M 

E6 

FJ17 

A 

7 

CA 

29 

M 

03 

FJ18 

A 

6 

NCA 

26 

M 

E4 

FJ19 

A 

21 

CA 

42 

M 

05 

FJ20 

A 

20 

CA 

41 

M 

05 

Table  49.  WRMSE  Model  Fit  Data  for  u  (v(x)) . 


Subj 

Linear 

Exponential 

Logarithmic 

Power 

Sigmoid  | 

WRMSE 

c 

WRMSE 

c 

WRMSE 

c 

WRMSE 

c 

WRMSE 

1 

FA 

0.075 

-0.794 

0.030 

1.83E+13 

0.070 

1.334 

0.024 

-0.277 

0.049 

Amb 

0.098 

0.223 

0.097 

3.19E+00 

0.096 

0.844 

0.093 

-0.251 

0.037 

Fuel 

0.182 

2.307 

0.125 

4.59E+02 

0.182 

0.449 

0.113 

0.267 

0.039 

Time 

0.133 

-1.463 

0.211 

4.49E+13 

0.114 

1.676 

0.219 

-0.306 

0.050 

2 

FA 

0.093 

0.884 

0.082 

6.01E-01 

0.080 

0.699 

0.076 

-0.394 

0.018 

Amb 

0.588 

112.967 

0.342 

1.14E+01 

0.585 

0.024 

0.324 

-0.537 

0.029 

Fuel 

0.275 

-1.757 

0.269 

6.53E+04 

0.275 

2.012 

0.269 

0.215 

0.022 

Time 

0.228 

-4.860 

0.079 

4.31E+1 1 

0.228 

4.641 

0.088 

-0.326 

0.064 

3 

FA 

0.067 

-0.208 

0.065 

1.55E+06 

0.067 

1.014 

0.067 

-0.257 

0.046 

Amb 

0.155 

2.245 

0.135 

7.40E-02 

0.119 

0.482 

0.105 

-0.307 

0.035 

Fuel 

0.193 

2.227 

0.155 

5.47E+01 

0.193 

0.430 

0.135 

0.291 

0.055 

Time 

0.266 

7.094 

0.103 

1.00E-02 

0.061 

0.307 

0.062 

-0.286 

0.062 

4 

FA 

0.091 

-0.959 

0.065 

9.15E+09 

0.091 

1.308 

0.075 

-0.261 

0.039 

Amb 

0.227 

5.796 

0.146 

3.79E+02 

0.227 

0.311 

0.107 

-0.256 

0.034 

Fuel 

0.111 

1.435 

0.073 

2.72E-01 

0.069 

0.596 

0.062 

0.341 

0.025 

Time 

0.185 

3.774 

0.113 

5.10E-02 

0.096 

0.459 

0.099 

-0.277 

0.043 

5 

FA 

0.153 

-1.960 

0.094 

4.62E+07 

0.153 

1.816 

0.110 

-0.287 

0.051 

Amb 

0.235 

5.963 

0.159 

3.63E+01 

0.234 

0.304 

0.119 

-0.243 

0.036 

Fuel 

0.092 

0.485 

0.088 

1.35E+00 

0.087 

0.782 

0.082 

0.266 

0.027 

Time 

0.189 

3.034 

0.148 

6.00E-02 

0.128 

0.479 

0.122 

-0.212 

0.062 

6 

FA 

0.188 

2.279 

0.147 

7.00E-02 

0.138 

0.422 

0.138 

-0.251 

0.043 

Amb 

0.410 

10.252 

0.365 

2.49E+01 

0.409 

0.130 

0.363 

-0.244 

0.038 

Fuel 

0.082 

-0.459 

0.075 

1.03E+06 

0.082 

1.100 

0.079 

0.248 

0.036 

Time 

0.131 

-1.744 

0.064 

2.91E+07 

0.131 

1.695 

0.079 

-0.326 

0.049 
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Subj 

Linear 

Exponential 

Logarithmic 

Power 

Sigmoid  | 

WRMSE 

c 

WRMSE 

c 

WRMSE 

c 

WRMSE 

c 

WRMSE 

7 

FA 

0.104 

-0.880 

0.089 

2.95E+09 

0.104 

1.244 

0.097 

-0.268 

0.041 

Amb 

0.211 

2.951 

0.141 

4.11E+02 

0.211 

0.370 

0.117 

-0.307 

0.056 

Fuel 

0.236 

-4.194 

0.153 

1.01E+1 1 

0.236 

3.773 

0.160 

0.330 

0.061 

Time 

0.126 

-0.554 

0.121 

1.02E+06 

0.126 

1.060 

0.125 

-0.254 

0.044 

8 

FA 

0.087 

0.459 

0.084 

1.41E+00 

0.083 

0.794 

0.079 

-0.266 

0.053 

Amb 

0.068 

-0.334 

0.064 

3.60E+09 

0.068 

1.048 

0.068 

-0.295 

0.041 

Fuel 

0.182 

2.650 

0.139 

5.10E-02 

0.122 

0.417 

0.111 

0.274 

0.054 

Time 

0.151 

-1.063 

0.142 

1.81E+05 

0.151 

1.311 

0.147 

-0.254 

0.072 

9 

FA 

0.098 

-0.892 

0.071 

1.50E+06 

0.098 

1.404 

0.070 

-0.258 

0.032 

Amb 

0.273 

4.346 

0.205 

3.40E+01 

0.272 

0.259 

0.184 

-0.326 

0.062 

Fuel 

0.122 

-1.672 

0.076 

2.81E+07 

0.122 

1.712 

0.086 

0.386 

0.058 

Time 

0.110 

-1.068 

0.082 

1.04E+1 1 

0.110 

1.347 

0.093 

-0.342 

0.101 

10 

FA 

0.146 

0.064 

0.147 

6.92E+00 

0.147 

0.873 

0.146 

-0.277 

0.032 

Amb 

0.117 

1.333 

0.066 

3.24E-01 

0.063 

0.627 

0.056 

-0.264 

0.032 

Fuel 

0.095 

0.753 

0.082 

8.57E-01 

0.081 

0.788 

0.080 

0.276 

0.037 

Time 

0.113 

1.271 

0.090 

8.42E+01 

0.113 

0.676 

0.081 

-0.266 

0.029 

11 

FA 

0.141 

-1.359 

0.114 

8.01E+05 

0.141 

1.416 

0.128 

-0.235 

0.067 

Amb 

0.232 

5.891 

0.152 

3.31E+01 

0.231 

0.307 

0.113 

-0.249 

0.038 

Fuel 

0.319 

7.490 

0.219 

2.95E+01 

0.318 

0.183 

0.195 

0.447 

0.098 

Time 

0.185 

4.231 

0.074 

4.30E-02 

0.058 

0.438 

0.072 

-0.328 

0.028 

12 

FA 

0.351 

10.819 

0.075 

7.32E-03 

0.113 

0.292 

0.150 

-0.313 

0.046 

Amb 

0.183 

2.980 

0.075 

5.80E-02 

0.039 

0.461 

0.041 

-0.280 

0.031 

Fuel 

0.193 

1.100 

0.190 

2.10E-01 

0.188 

0.550 

0.184 

0.275 

0.047 

Time 

0.237 

1.229 

0.229 

1.24E-01 

0.218 

0.527 

0.202 

-0.085 

0.163 

13 

FA 

0.131 

-0.399 

0.130 

7.81E+03 

0.131 

0.976 

0.131 

-0.271 

0.039 

Amb 

0.106 

-0.895 

0.091 

7.12E+09 

0.106 

1.272 

0.098 

-0.267 

0.041 

Fuel 

0.089 

-1.121 

0.061 

6.54E+06 

0.089 

1.373 

0.074 

0.350 

0.061 

Time 

0.109 

-0.534 

0.104 

5.03E+05 

0.109 

1.033 

0.109 

-0.242 

0.049 

14 

FA 

0.135 

0.073 

0.135 

6.38E+00 

0.135 

0.862 

0.134 

-0.329 

0.037 

Amb 

0.323 

5.069 

0.271 

3.13E+01 

0.322 

0.232 

0.260 

-0.256 

0.041 

Fuel 

0.092 

0.952 

0.068 

5.78E-01 

0.066 

0.702 

0.060 

0.295 

0.030 

Time 

0.178 

3.984 

0.128 

3.90E-02 

0.102 

0.449 

0.099 

-0.253 

0.034 

15 

FA 

0.367 

6.729 

0.016 

1.50E+02 

0.366 

0.319 

0.135 

-0.254 

0.037 

Amb 

0.325 

6.428 

0.054 

1.93E+02 

0.325 

0.327 

0.106 

-0.285 

0.051 

Fuel 

0.190 

1.044 

0.186 

2.29E-01 

0.184 

0.562 

0.180 

0.364 

0.064 

Time 

0.293 

6.713 

0.065 

1.10E-01 

0.148 

0.302 

0.088 

-0.460 

0.065 

16 

FA 

0.145 

2.585 

0.056 

1.05E-01 

0.040 

0.544 

0.053 

-0.347 

0.023 

Amb 

0.320 

5.224 

0.088 

1.85E+02 

0.320 

0.394 

0.160 

0.005 

0.356 

Fuel 

0.095 

1.204 

0.068 

4.44E-01 

0.065 

0.721 

0.063 

0.326 

0.059 

Time 

0.135 

1.143 

0.116 

3.19E-01 

0.111 

0.602 

0.098 

-0.251 

0.082 

17 

FA 

0.213 

4.008 

0.112 

1 .80E-02 

0.055 

0.376 

0.054 

-0.302 

0.047 

Amb 

0.169 

2.536 

0.079 

4.48E+02 

0.169 

0.465 

0.030 

-0.314 

0.050 

Fuel 

0.366 

19.519 

0.080 

1.63E+02 

0.366 

0.256 

0.161 

0.335 

0.033 

Time 

0.280 

3.625 

0.217 

7.44E+01 

0.280 

0.287 

0.186 

-0.245 

0.115 

18 

FA 

0.152 

-1.709 

0.058 

3.25E+07 

0.152 

1.851 

0.049 

-0.316 

0.037 

Amb 

0.326 

-5.269 

0.140 

6.10E+10 

0.326 

4.925 

0.150 

-0.391 

0.019 

Fuel 

0.231 

3.446 

0.087 

3.40E-02 

0.037 

0.381 

0.037 

0.303 

0.081 

Time 

0.115 

-1.377 

0.052 

1.54E+07 

0.115 

1.603 

0.068 

-0.194 

0.055 

19 

FA 

0.082 

-0.890 

0.052 

2.37E+13 

0.073 

1.309 

0.060 

-0.247 

0.043 

Amb 

0.354 

7.302 

0.317 

3.47E+01 

0.353 

0.043 

0.305 

-0.296 

0.051 

Fuel 

0.322 

7.665 

0.042 

1.97E+02 

0.322 

0.272 

0.110 

0.486 

0.069 

Time 

0.252 

5.780 

0.061 

2.57E+02 

0.252 

0.366 

0.079 

-0.327 

0.023 

20 

FA 

0.186 

3.446 

0.075 

4.80E-02 

0.037 

0.452 

0.046 

-0.282 

0.040 

Amb 

0.286 

7.755 

0.101 

2.69E+02 

0.286 

0.316 

0.085 

-0.303 

0.062 

Fuel 

0.179 

3.996 

0.130 

4.75E+01 

0.178 

0.451 

0.101 

0.272 

0.048 

Time 

0.220 

6.813 

0.111 

3.22E+01 

0.219 

0.364 

0.088 

-0.294 

0.042 
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Table  50.  RMSE  Model  Fit  Data  for  u (v (jc)) . 


Subj 

Linear 

Exponential 

Logarithmic 

Power 

Sigmoid  | 

RMSE 

c 

RMSE 

c 

RMSE 

c 

RMSE 

c 

RMSE 

1 

FA 

0.082 

-0.794 

0.034 

1.830E+13 

0.078 

1.334 

0.027 

-0.277 

0.046 

Amb 

0.101 

0.223 

0.100 

3.189E+00 

0.100 

0.844 

0.096 

-0.251 

0.034 

Fuel 

0.206 

2.307 

0.139 

4.589E+02 

0.206 

0.449 

0.123 

0.267 

0.037 

Time 

0.150 

-1.463 

0.239 

4.493E+13 

0.120 

1.676 

0.251 

-0.306 

0.052 

2 

FA 

0.104 

0.884 

0.090 

6.010E-01 

0.088 

0.699 

0.082 

-0.394 

0.015 

Amb 

0.612 

112.967 

0.390 

1.137E+01 

0.609 

0.024 

0.370 

-0.537 

0.031 

Fuel 

0.315 

-1.757 

0.309 

6.530E+04 

0.315 

2.012 

0.309 

0.215 

0.020 

Time 

0.258 

-4.860 

0.089 

4.310E+1 1 

0.258 

4.641 

0.100 

-0.326 

0.053 

3 

FA 

0.072 

-0.208 

0.070 

1.550E+06 

0.072 

1.014 

0.072 

-0.257 

0.040 

Amb 

0.173 

2.245 

0.143 

7.400E-02 

0.123 

0.482 

0.108 

-0.307 

0.034 

Fuel 

0.214 

2.227 

0.164 

5.466E+01 

0.213 

0.430 

0.141 

0.291 

0.045 

Time 

0.296 

7.094 

0.111 

1.000E-02 

0.067 

0.307 

0.066 

-0.286 

0.056 

4 

FA 

0.100 

-0.959 

0.070 

9.153E+09 

0.100 

1.308 

0.081 

-0.261 

0.035 

Amb 

0.256 

5.796 

0.156 

3.792E+02 

0.255 

0.311 

0.111 

-0.256 

0.032 

Fuel 

0.125 

1.435 

0.078 

2.720E-01 

0.073 

0.596 

0.065 

0.341 

0.022 

Time 

0.212 

3.774 

0.126 

5.100E-02 

0.105 

0.459 

0.105 

-0.277 

0.037 

5 

FA 

0.167 

-1.960 

0.099 

4.622E+07 

0.167 

1.816 

0.116 

-0.287 

0.043 

Amb 

0.264 

5.963 

0.169 

3.629E+01 

0.263 

0.304 

0.124 

-0.243 

0.034 

Fuel 

0.100 

0.485 

0.095 

1.353E+00 

0.094 

0.782 

0.087 

0.266 

0.023 

Time 

0.211 

3.034 

0.164 

6.000E-02 

0.141 

0.479 

0.132 

-0.212 

0.056 

6 

FA 

0.217 

2.279 

0.167 

7.000E-02 

0.156 

0.422 

0.146 

-0.251 

0.040 

Amb 

0.440 

10.252 

0.403 

2.494E+01 

0.440 

0.130 

0.402 

-0.244 

0.036 

Fuel 

0.090 

-0.459 

0.084 

1.029E+06 

0.090 

1.100 

0.088 

0.248 

0.029 

Time 

0.146 

-1.744 

0.069 

2.910E+07 

0.146 

1.695 

0.085 

-0.326 

0.042 

7 

FA 

0.135 

-0.554 

0.131 

1.022E+06 

0.135 

1.060 

0.135 

-0.254 

0.035 

Amb 

0.235 

2.951 

0.150 

4.111E+02 

0.235 

0.370 

0.122 

-0.307 

0.054 

Fuel 

0.267 

-4.194 

0.165 

1.005E+1 1 

0.267 

3.773 

0.173 

0.330 

0.053 

Time 

0.135 

-0.554 

0.131 

1.022E+06 

0.135 

1.060 

0.135 

-0.254 

0.037 

8 

FA 

0.095 

0.459 

0.090 

1.414E+00 

0.089 

0.794 

0.083 

-0.266 

0.053 

Amb 

0.070 

-0.334 

0.066 

3.602E+09 

0.070 

1.048 

0.070 

-0.295 

0.040 

Fuel 

0.207 

2.650 

0.151 

5.100E-02 

0.131 

0.417 

0.117 

0.274 

0.048 

Time 

0.163 

-1.063 

0.151 

1.813E+05 

0.163 

1.311 

0.157 

-0.254 

0.068 

9 

FA 

0.107 

-0.892 

0.073 

1.503E+06 

0.107 

1.404 

0.071 

-0.258 

0.026 

Amb 

0.308 

4.346 

0.229 

3.399E+01 

0.307 

0.259 

0.206 

-0.326 

0.061 

Fuel 

0.137 

-1.672 

0.079 

2.808E+07 

0.137 

1.712 

0.091 

0.386 

0.059 

Time 

0.117 

-1.068 

0.091 

1.035E+1 1 

0.117 

1.347 

0.102 

-0.342 

0.080 

10 

FA 

0.166 

0.064 

0.166 

6.923E+00 

0.166 

0.873 

0.165 

-0.277 

0.025 

Amb 

0.123 

1.333 

0.068 

3.240E-01 

0.063 

0.627 

0.054 

-0.264 

0.031 

Fuel 

0.104 

0.753 

0.089 

8.570E-01 

0.087 

0.788 

0.086 

0.276 

0.031 

Time 

0.125 

1.271 

0.098 

8.418E+01 

0.125 

0.676 

0.087 

-0.266 

0.024 

11 

FA 

0.154 

-1.359 

0.123 

8.014E+05 

0.154 

1.416 

0.137 

-0.235 

0.057 

Amb 

0.260 

5.891 

0.163 

3.309E+01 

0.259 

0.307 

0.118 

-0.249 

0.037 

Fuel 

0.356 

7.490 

0.232 

2.950E+01 

0.354 

0.183 

0.441 

0.447 

0.080 

Time 

0.214 

4.231 

0.084 

4.300E-02 

0.064 

0.438 

0.073 

-0.328 

0.025 

12 

FA 

0.394 

10.819 

0.081 

7.316E-03 

0.122 

0.292 

0.155 

-0.313 

0.047 

Amb 

0.204 

2.980 

0.082 

5.800E-02 

0.043 

0.461 

0.038 

-0.280 

0.029 

Fuel 

0.222 

1.100 

0.217 

2.100E-01 

0.215 

0.550 

0.210 

0.275 

0.044 

Time 

0.234 

1.229 

0.219 

1.240E-01 

0.205 

0.527 

0.186 

-0.085 

0.133 

13 

FA 

0.144 

-0.399 

0.143 

7.805E+03 

0.144 

0.976 

0.144 

-0.271 

0.034 

Amb 

0.113 

-0.895 

0.095 

7.120E+09 

0.113 

1.272 

0.104 

-0.267 

0.039 

Fuel 

0.101 

-1.121 

0.067 

6.540E+06 

0.101 

1.373 

0.081 

0.350 

0.060 

Time 

0.117 

-0.534 

0.112 

5.025E+05 

0.117 

1.033 

0.117 

-0.242 

0.042 

14 

FA 

0.149 

0.073 

0.149 

6.384E+00 

0.149 

0.862 

0.147 

-0.329 

0.032 

Amb 

0.360 

5.069 

0.306 

3.130E+01 

0.359 

0.232 

0.294 

-0.256 

0.039 

Fuel 

0.102 

0.952 

0.073 

5.780E-01 

0.070 

0.702 

0.064 

0.295 

0.024 

Time 

0.203 

3.984 

0.143 

3.900E-02 

0.112 

0.449 

0.106 

-0.253 

0.028 

15 

FA 

0.392 

6.729 

0.015 

1.502E+02 

0.392 

0.319 

0.130 

-0.254 

0.033 

Amb 

0.354 

6.428 

0.057 

1.928E+02 

0.354 

0.327 

0.100 

-0.285 

0.049 

Fuel 

0.209 

1.044 

0.205 

2.290E-01 

0.203 

0.562 

0.199 

0.364 

0.056 

Time 

0.333 

6.713 

0.074 

1.100E-01 

0.167 

0.302 

0.077 

-0.460 

0.055 
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Subj 

Linear 

Exponential 

Logarithmic 

Power 

Sigmoid  | 

RMSE 

c 

RMSE 

c 

RMSE 

c 

RMSE 

c 

RMSE 

16 

FA 

0.168 

2.585 

0.060 

1.050E-01 

0.042 

0.544 

0.055 

-0.347 

0.020 

Amb 

0.340 

5.224 

0.094 

1.850E+02 

0.340 

0.394 

0.162 

0.005 

0.328 

Fuel 

0.107 

1.204 

0.071 

4.440E-01 

0.067 

0.721 

0.065 

0.326 

0.055 

Time 

0.147 

1.143 

0.122 

3.190E-01 

0.116 

0.602 

0.102 

-0.251 

0.072 

17 

FA 

0.244 

4.008 

0.126 

1.800E-02 

0.062 

0.376 

0.053 

-0.302 

0.043 

Amb 

0.188 

2.536 

0.084 

4.480E+02 

0.188 

0.465 

0.028 

-0.314 

0.049 

Fuel 

0.407 

19.519 

0.087 

1.628E+02 

0.406 

0.256 

0.167 

0.335 

0.033 

Time 

0.311 

3.625 

0.230 

7.437E+01 

0.311 

0.287 

0.193 

-0.245 

0.091 

18 

FA 

0.172 

-1.709 

0.064 

3.254E+07 

0.172 

1.851 

0.054 

-0.316 

0.037 

Amb 

0.312 

-5.269 

0.110 

6.103E+10 

0.312 

4.925 

0.124 

-0.391 

0.070 

Fuel 

0.252 

3.446 

0.090 

3.400E-02 

0.041 

0.381 

0.032 

0.303 

0.065 

Time 

0.130 

-1.377 

0.048 

1.543E+07 

0.130 

1.603 

0.064 

-0.194 

0.048 

19 

FA 

0.090 

-0.890 

0.056 

2.368E+13 

0.079 

1.309 

0.064 

-0.247 

0.038 

Amb 

0.395 

7.302 

0.358 

3.470E+01 

0.395 

0.043 

0.345 

-0.296 

0.052 

Fuel 

0.368 

7.665 

0.048 

1.975E+02 

0.368 

0.272 

0.099 

0.486 

0.057 

Time 

0.288 

5.780 

0.069 

2.572E+02 

0.288 

0.366 

0.078 

-0.327 

0.018 

20 

FA 

0.212 

3.446 

0.084 

4.800E-02 

0.042 

0.452 

0.045 

-0.282 

0.036 

Amb 

0.319 

7.755 

0.109 

2.688E+02 

0.319 

0.316 

0.079 

-0.303 

0.057 

Fuel 

0.207 

3.996 

0.145 

4.746E+01 

0.206 

0.451 

0.109 

0.272 

0.045 

Time 

0.253 

6.813 

0.125 

3.220E+01 

0.252 

0.364 

0.090 

-0.294 

0.035 

Table  51.  Summary  of  Model  Fits  for  w(v(x))  Employing  RMSE.  “Lin”  stands  for 
linear,  “Log”  for  logarithmic,  “Pow”  for  power  functions. 


Subject 

Best 

Acceptable  Fit 

G 

Other 

Acceptable 

Curve 

Shape 

1 

FA 

Power 

0.90 

Lin,  Exp,  Log,Sig 

Near  Linear 

Amb 

Sigmoid 

0.89 

Lin,  Exp, 

Log, Pow 

S-Shape 

Fuel 

Sigmoid 

0.97 

None 

Concave 

Time 

Sigmoid 

0.88 

None 

Concave 

2 

FA 

Sigmoid 

0.98 

Lin,  Exp, 

Log, Pow 

S-Shape 

Amb 

Sigmoid 

1.00 

None 

Concave 

Fuel 

Sigmoid 

1.00 

None 

S-Shape 

Time 

Sigmoid 

0.88 

Exp, Power 

S-Shape 

3 

FA 

Sigmoid 

0.69 

Lin, Exp, Log, Pow 

Near  Linear 

Amb 

Sigmoid 

0.96 

Pow 

Concave 

Fuel 

Sigmoid 

0.96 

None 

Concave 

Time 

Sigmoid 

0.96 

Exp,  Log,  Pow 

S-Shape 

4 

FA 

Sigmoid 

0.50 

Lin, Exp, Log, Pow 

Convex 

Amb 

Sigmoid 

0.98 

Power 

Concave 

Fuel 

Sigmoid 

0.97 

Exp, Log, Pow 

Concave 

Time 

Sigmoid 

0.97 

Log, Pow 

S-Shape 

5 

FA 

Sigmoid 

0.94 

Exp, Power 

Convex 

Amb 

Sigmoid 

0.98 

None 

Concave 
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Subject 

Best 

Acceptable  Fit 

G 

Other 

Acceptable 

Curve 

Shape 

Fuel 

Sigmoid 

0.95 

Lin, Exp, Log, Pow 

Concave 

Time 

Sigmoid 

0.61 

None 

S-Shape 

6 

FA 

Sigmoid 

0.97 

None 

Concave 

Amb 

Sigmoid 

0.99 

None 

Concave 

Fuel 

Sigmoid 

0.90 

Lin, Exp, Log, Pow 

Near  Linear 

Time 

Sigmoid 

0.92 

Exp, Power 

Convex 

7 

FA 

Sigmoid 

0.90 

Lin, Exp, Log, Pow 

S-Shape 

Amb 

Sigmoid 

0.95 

None 

Concave 

Fuel 

Sigmoid 

0.96 

None 

S-Shape 

Time 

Sigmoid 

0.93 

None 

S-Shape 

8 

FA 

Sigmoid 

0.69 

Lin, Exp, Log, Pow 

Concave 

Amb 

Sigmoid 

0.68 

Lin, Exp, Log, Pow 

Near  Linear 

Fuel 

Sigmoid 

0.95 

Pow 

Concave 

Time 

Sigmoid 

0.83 

None 

S-Shape 

9 

FA 

Sigmoid 

0.94 

Lin, Exp, Log, Pow 

S-Shape 

Amb 

Sigmoid 

0.96 

None 

Concave 

Fuel 

Sigmoid 

0.82 

Exp, Pow 

S-Shape 

Time 

Sigmoid 

0.53 

Lin, Exp, Log, Pow 

S-Shape 

10 

FA 

Sigmoid 

0.98 

None 

S-Shape 

Amb 

Sigmoid 

0.94 

Exp,  Log, Pow 

Concave 

Fuel 

Sigmoid 

0.91 

Lin,  Exp, 

Log, Pow 

S-Shape 

Time 

Sigmoid 

0.96 

Exp,  Pow 

S-Shape 

11 

FA 

Sigmoid 

0.86 

None 

S-Shape 

Amb 

Sigmoid 

0.98 

Power 

Concave 

Fuel 

Sigmoid 

0.95 

None 

Concave 

Time 

Sigmoid 

0.99 

Exp, Log, Pow 

S-Shape 

12 

FA 

Sigmoid 

0.99 

Exp 

Concave 

Amb 

Sigmoid 

0.98 

Exp, Log, Pow 

Concave 

Fuel 

Sigmoid 

0.96 

None 

S-Shape 

Time 

None 

0.68 

None 

S-Shape 

13 

FA 

Sigmoid 

0.94 

None 

S-Shape 

Amb 

Sigmoid 

0.88 

Lin, Exp, Log, Pow 

S-Shape 

Fuel 

Sigmoid 

0.65 

Lin, Exp, Log, Pow 

Concave 

Time 

Sigmoid 

0.87 

Lin, Exp, Log, Pow 

S-Shape 

14 

FA 

Sigmoid 

0.95 

None 

S-Shape 

Amb 

Sigmoid 

0.99 

None 

Concave 

Fuel 

Sigmoid 

0.94 

Lin,  Exp, 

Log, Pow 

Concave 

Time 

Sigmoid 

0.98 

Log, Pow 

S-Shape 

15 

FA 

Expo 

1.00 

Sig 

Concave 
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Subject 

Best 

Acceptable  Fit 

G 

Other 

Acceptable 

Curve 

Shape 

Amb 

Sigmoid 

0.98 

Exp, Power 

Concave 

Fuel 

Sigmoid 

0.93 

None 

S-Shape 

Time 

Sigmoid 

0.97 

Exp, Power 

Concave 

16 

FA 

Sigmoid 

0.99 

Exp, Log, Pow 

Concave 

Amb 

Expo 

0.92 

None 

Concave 

Fuel 

Sigmoid 

0.73 

Lin,  Exp, 

Log, Pow 

S-Shape 

Time 

Sigmoid 

0.76 

Log, Pow 

Concave 

17 

FA 

Sigmoid 

0.97 

Log, Pow 

Concave 

Amb 

Power 

0.98 

Exp,Sig 

Concave 

Fuel 

Sigmoid 

0.99 

Exp 

Concave 

Time 

Sigmoid 

0.91 

None 

Concave 

18 

FA 

Sigmoid 

0.95 

Exp, Power 

S-Shape 

Amb 

Sigmoid 

0.95 

Exp 

Convex 

Fuel 

Power 

0.98 

Exp,Log,Sig 

Concave 

Time 

Sigmoid 

0.87 

Exp, Pow 

S-Shape 

19 

FA 

Sigmoid 

0.82 

Lin, Exp, Log, Pow 

Near  Linear 

Amb 

Sigmoid 

0.98 

None 

Concave 

Fuel 

Expo 

0.98 

Power, Sig 

Concave 

Time 

Sigmoid 

1.00 

Exp, Power 

Concave 

20 

FA 

Sigmoid 

0.97 

Exp, Log, Pow 

Concave 

Amb 

Sigmoid 

0.97 

Exp, Power 

Concave 

Fuel 

Sigmoid 

0.95 

Power 

S-Shape 

Time 

Sigmoid 

0.98 

Exp, Power 

S-Shape 

Table  52.  Summary  of  Model  Fits  for  w”odeI  (v(>  (x))  Employing  WRMSE.  “Lin”  stands 
for  linear,  “Log”  for  logarithmic,  “Pow”  for  power  functions. 


Subject 

Best 

Acceptable  Fit 

G 

Other 

Acceptable 

Curve 

Shape 

1 

FA 

Power 

0.9007 

Lin, Exp, Log, Sig 

Near 

Linear 

Amb 

Sigmoid 

0.8601 

Lin, Exp, Log, Pow 

S-Shape 

Fuel 

Sigmoid 

0.9538 

None 

Concave 

Time 

Sigmoid 

0.8588 

None 

Concave 

2 

FA 

Sigmoid 

0.9607 

Lin, Exp, Log, Pow 

S-Shape 
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Subject 

Best 

Acceptable  Fit 

G 

Other 

Acceptable 

Curve 

Shape 

Amb 

Sigmoid 

0.9975 

None 

Concave 

Fuel 

Sigmoid 

0.9934 

None 

S-Shape 

Time 

Sigmoid 

0.9216 

Exp,Pow 

S-Shape 

3 

FA 

Sigmoid 

0.5345 

Lin, Exp, Log, Pow 

Near 

Linear 

Amb 

Sigmoid 

0.9483 

Power 

Concave 

Fuel 

Sigmoid 

0.9198 

None 

Concave 

Time 

Log 

0.9470 

Exp,Pow,Sig 

S-Shape 

4 

FA 

Sigmoid 

0.8111 

Lin, Exp, Log, Pow 

Convex 

Amb 

Sigmoid 

0.9775 

Pow 

Concave 

Fuel 

Sigmoid 

0.9510 

Exp, Log, Pow 

Concave 

Time 

Sigmoid 

0.9456 

Log, Pow 

S-Shape 

5 

FA 

Sigmoid 

0.8901 

Exp 

Convex 

Amb 

Sigmoid 

0.9763 

None 

Concave 

Fuel 

Sigmoid 

0.9117 

Lin, Exp, Log, Pow 

Concave 

Time 

Sigmoid 

0.8903 

None 

S-Shape 

6 

FA 

Sigmoid 

0.9473 

None 

Concave 

Amb 

Sigmoid 

0.9915 

None 

Concave 

Fuel 

Sigmoid 

0.8095 

Lin, Exp, Log, Pow 

Near 

Linear 

Time 

Sigmoid 

0.8584 

Exp, Pow 

Convex 

7 

FA 

Sigmoid 

0.8444 

Lin, Exp, Log, Pow 

S-Shape 

Amb 

Sigmoid 

0.9295 

None 

Concave 

Fuel 

Sigmoid 

0.9334 

None 

S-Shape 

Time 

Sigmoid 

0.8756 

None 

S-Shape 

8 

FA 

Sigmoid 

0.6219 

Lin, Exp, Log, Pow 

Concave 

Amb 

Sigmoid 

0.6345 

Lin, Exp, Log, Pow 

Near 

Linear 

Fuel 

Sigmoid 

0.9107 

None 

Concave 

Time 

Sigmoid 

0.7692 

None 

S-Shape 

9 

FA 

Sigmoid 

0.8938 

Lin, Exp, Log, Pow 

S-Shape 

Amb 

Sigmoid 

0.9494 

None 

Concave 

Fuel 

Sigmoid 

0.7740 

Exp, Pow 

S-Shape 

Time 

Expo 

0.4439 

Pow,Sig 

S-Shape 

10 

FA 

Sigmoid 

0.9530 

None 

S-Shape 

Amb 

Sigmoid 

0.9246 

Exp, Log, Pow 

Concave 

Fuel 

Sigmoid 

0.8460 

Lin, Exp, Log, Pow 

S-Shape 

Time 

Sigmoid 

0.9356 

Exp, Pow 

S-Shape 

11 

FA 

Sigmoid 

0.7766 

None 

S-Shape 

Amb 

Sigmoid 

0.9726 

None 

Concave 

Fuel 

Sigmoid 

0.9051 

None 

Concave 
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Subject 

Best 

Acceptable  Fit 

G 

Other 

Acceptable 

Curve 

Shape 

Time 

Sigmoid 

0.9770 

Exp,Log,Pow 

S-Shape 

12 

FA 

Sigmoid 

0.9830 

Exp 

Concave 

Amb 

Sigmoid 

0.9711 

Exp,Log,Pow 

Concave 

Fuel 

Sigmoid 

0.9393 

None 

S-Shape 

Time 

None 

0.5240 

None 

S-Shape 

13 

FA 

Sigmoid 

0.9109 

None 

S-Shape 

Amb 

Sigmoid 

0.8477 

Lin, Exp, Log, Pow 

S-Shape 

Fuel 

Expo 

0.5359 

Lin,Log,Pow,Sig 

Concave 

Time 

Sigmoid 

0.7950 

Exp 

S-Shape 

14 

FA 

Sigmoid 

0.9233 

None 

S-Shape 

Amb 

Sigmoid 

0.9841 

None 

Concave 

Fuel 

Sigmoid 

0.8953 

Lin, Exp, Log, Pow 

Concave 

Time 

Sigmoid 

0.9635 

Log, Pow 

S-Shape 

15 

FA 

Expo 

0.9981 

Sig 

Concave 

Amb 

Sigmoid 

0.9751 

Exp, Pow 

Concave 

Fuel 

Sigmoid 

0.8877 

None 

S-Shape 

Time 

Sigmoid 

0.9512 

Exp, Pow 

Concave 

16 

FA 

Sigmoid 

0.9747 

Exp, Log, Pow 

Concave 

Amb 

Expo 

0.9248 

None 

Concave 

Fuel 

Sigmoid 

0.6097 

Lin, Exp, Log, Pow 

S-Shape 

Time 

Sigmoid 

0.6289 

Pow 

Concave 

17 

FA 

Sigmoid 

0.9516 

Log, Pow 

Concave 

Amb 

Power 

0.9680 

Exp, Sig 

Concave 

Fuel 

Sigmoid 

0.9918 

Exp 

Concave 

Time 

None 

0.8318 

None 

Concave 

18 

FA 

Sigmoid 

0.9394 

Exp, Pow 

S-Shape 

Amb 

Sigmoid 

0.9968 

None 

Convex 

Fuel 

Power 

0.9748 

Exp, Log, Sig 

Concave 

Time 

Expo 

0.7959 

Pow, Sig 

S-Shape 

19 

FA 

Sigmoid 

0.7212 

Lin, Exp, Log, Pow 

Near 

Linear 

Amb 

Sigmoid 

0.9796 

None 

Concave 

Fuel 

Expo 

0.9830 

Sig 

Concave 

Time 

Sigmoid 

0.9919 

Exp, Pow 

Concave 

20 

FA 

Log 

0.9604 

Exp, Pow, Sig 

Concave 

Amb 

Sigmoid 

0.9532 

Exp, Power 

Concave 

Fuel 

Sigmoid 

0.9285 

Power 

S-Shape 

Time 

Sigmoid 

0.9637 

Power 

S-Shape 
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The  results  of  the  multidimensional  analysis  of  the  tactical  problem  are  tabulated 


in  Table  53.  The  results  are  listed  by  subject  by  route  for  the  various  preference 
functions:  value  function,  v(x)  ;  utility  function  determined  by  the  certainty-equivalent 

method,  uCE  (x) ;  utility  function  determined  by  the  probability-equivalent  method, 

uPE  (x) ;  utility  function  determined  by  averaging  the  certainty-equivalent  and 

probability-equivalent  methods,  u  (x) ;  and  Kirkwood’s  (1997)  multiattribute  utility 

method,  um  (x) .  The  last  function  has  subscripts  of  one  and  two.  Two  functions  are 

present  for  those  subjects  who  were  never  ambivalent  between  the  alternatives  presented. 
The  two  functions  bound  the  ambivalence  of  the  subject. 


Table  53.  Route  Alternative  Mathematical  Expectation  Scores  for  Preference  Functions. 


Subject 

Route 

v(x) 

uCE  (*) 

uPE  (x) 

u  (x) 

Unh  (*) 

um2  M 

FJ1 

A 

0.404 

0.487 

0.523 

0.505 

0.501 

0.414 

B 

0.254 

0.341 

0.372 

0.356 

0.335 

0.262 

C 

0.180 

0.263 

0.303 

0.280 

0.244 

0.185 

D 

0.218 

0.291 

0.344 

0.317 

0.290 

0.225 

E 

0.679 

0.673 

0.703 

0.688 

0.762 

0.688 

F 

0.610 

0.613 

0.660 

0.637 

0.702 

0.620 

FJ2 

A 

0.182 

0.462 

0.461 

0.439 

0.021 

- 

B 

0.144 

0.313 

0.230 

0.271 

0.016 

- 

C 

0.044 

0.250 

0.171 

0.207 

0.004 

- 

D 

0.073 

0.308 

0.271 

0.290 

0.007 

- 

E 

0.487 

0.671 

0.773 

0.722 

0.123 

- 

F 

0.294 

0.616 

0.642 

0.629 

0.052 

- 

FJ3 

A 

0.470 

0.588 

0.621 

0.605 

0.307 

- 

B 

0.398 

0.411 

0.491 

0.451 

0.248 

- 

C 

0.220 

0.268 

0.407 

0.336 

0.122 

- 

D 

0.150 

0.249 

0.361 

0.305 

0.080 

- 

E 

0.539 

0.516 

0.556 

0.536 

0.373 

- 

F 

0.486 

0.496 

0.562 

0.529 

0.326 

- 

FJ4 

A 

0.282 

0.497 

0.419 

0.458 

0.274 

0.544 
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Subject 

Route 

vW 

« CE  (*) 

uPE  (x) 

u  (x) 

Unh  (*) 

um2  (*) 

B 

0.341 

0.366 

0.309 

0.338 

0.714 

0.171 

C 

0.371 

0.351 

0.337 

0.323 

0.762 

0.186 

D 

0.463 

0.404 

0.428 

0.416 

0.847 

0.253 

E 

0.856 

0.745 

0.833 

0.789 

0.985 

0.719 

F 

0.780 

0.693 

0.770 

0.732 

0.972 

0.602 

FJ19 

A 

0.387 

0.618 

0.616 

0.617 

0.670 

- 

B 

0.341 

0.387 

0.462 

0.425 

0.616 

- 

C 

0.215 

0.264 

0.385 

0.322 

0.442 

- 

D 

0.150 

0.294 

0.389 

0.342 

0.329 

- 

E 

0.579 

0.613 

0.642 

0.627 

0.824 

- 

F 

0.543 

0.606 

0.654 

0.630 

0.796 

- 

FJ20 

A 

0.215 

0.447 

0.459 

0.453 

0.453 

- 

B 

0.128 

0.305 

0.326 

0.315 

0.315 

- 

C 

0.055 

0.251 

0.295 

0.267 

0.267 

- 

D 

0.076 

0.346 

0.374 

0.360 

0.360 

- 

E 

0.730 

0.760 

0.756 

0.758 

0.758 

- 

F 

0.541 

0.658 

0.684 

0.671 

0.671 

- 

The  route  alternatives  ordered  in  descending  preference  appear  in  Table  54. 


Table  54.  Mathematical  Expectation  Ordered  Route  Alternatives  for  Preference 

Functions. 


Subject 

Order 

v(9 

uCE(x) 

uPE  (x) 

u  (x) 

Unh  (*) 

Um2  M 

FJ1 

1 

E 

E 

E 

E 

E 

E 

2 

F 

F 

F 

F 

F 

F 

3 

A 

A 

A 

A 

A 

A 

4 

B 

B 

B 

B 

B 

B 

5 

D 

D 

D 

D 

D 

D 

6 

C 

C 

C 

C 

C 

C 

FJ2 

1 

E 

E 

E 

E 

E 

2 

F 

F 

F 

F 

F 

3 

A 

A 

A 

A 

A 

4 

B 

B 

D 

D 

B 

5 

D 

D 

B 

B 

D 

6 

C 

C 

C 

C 

C 

FJ3 

1 

E 

A 

A 

A 

E 

2 

F 

E 

F 

E 

F 
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Appendix  C.  Information  Operations  Detailed  Results 


Information  Operations  Value  and  Utility  Functions. 
Attack  Information  Realm  Hierarchy.  Attack  Information. 


Figure  113.  Increase  Minimum  Update  Goal. 


1—1 

High  Med  Low  None 


Figure  115.  Likely  Accept  False  Goal. 
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□  Value 

□  Utility 


Figure  1 16.  Likely  Accept  True  Reject  False  Goal. 


Figure  117.  Likely  Reject  Truth  Goal. 
Attack  Information  Systems. 


10  - 

Q 

b 

4  - 

□ 

□ 

o 

o  - 

Stop  Great  Mod  None 

Figure  118.  Reduce  Bandwidth  Goal. 
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10  - 

D 

4 

□  Value 

□  Utility 

9  _ 

o  - 

Stop  Great  Mod  None 

Figure  1 19.  Reduce  Throughput  Goal. 


Value 

Utility 


Figure  120.  Degrade  Update  Goal. 


Figure  121.  Increase  Error  Content  Goal. 
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10  - 

Q 

D 

4 

□  Value 

□  Utility 

9  _ 

o  - 

Complete  Mod  Slight  None 

Figure  122.  Penetrate  System  Goal. 


Figure  123.  Increase  Recovery  Time  Goal. 


10  - 
8 

Defeat  Security  Goal 

R 

□  Value 

□ 

A 

9 

0  - 

High  Mod  Low  No 

Change 

Figure  124.  Defeat  Security  Goal. 
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10  - 

Q 

D 

4 

□  Value 

□  Utility 

9 

o  - 

Low  Med  High  Certain 

Figure  125.  Defeat  Detection  Goal. 


Attack  Information-Based  Processes. 


Value 

Utility 


Figure  126.  Consume  Essential  Resources  Goal. 


Value 

Utility 


Figure  127.  Consume  Essential  Time  Goal. 
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Value 

Utility 


Figure  129.  Decrease  Accuracy  Goal. 


UValue 
□  Utility 


Complete  Mod 


Slight 


None 


Figure  130.  Reduce  Focus  Goal. 
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10 

8 

6 

4 

2 

0 


□  Value 

□  Utility 


Catastrophic 


Slight 


Figure  131.  Reduce  Resilience  Goal. 


Minimize  Cost  Hierarchy. 


Figure  132.  Friendly  at  Risk  Cost. 


Value 

Utility 


Figure  133.  Maximum  System  Survivability  Cost. 
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Value 

Utility 


Figure  134.  Minimize  Collateral  Damage. 


Value 

Utility 


Figure  135.  Sensitive  Information  Security  Cost. 


Table  55.  WRMSE  Model  Fit  Data  for  u  (v(x)) . 


Linear 

Exponential 

Logarithmic 

Power 

Sigmoid  | 

WRMSE 

c 

WRMSE 

c 

WRMSE 

c 

WRMSE 

c 

WRMSE 

Increase 

Recovery 

Time 

0.00406 

-0.045 

0.00310 

2.58E+08 

0.00406 

1.014 

0.00353 

0.528 

0.0612 

Degrade 

Update 

0.122 

0.516 

0.111 

1.628 

0.112 

0.871 

0.117 

0.292 

0.0402 

Increase 

Error 

Content 

0.182 

-3.745 

0.0682 

1.31E+1 1 

0.182 

3.396 

0.0817 

0.437 

0.0588 

Age 

Devaluation 

0.166 

-3.556 

0.0661 

8.08E+10 

0.166 

3.21 

0.0786 

0.529 

0.0611 

Decrease 

Accuracy 

0.143 

-0.962 

0.109 

2661000 

0.143 

1.421 

0.0986 

0.336 

0.0478 

Reduce 

Timeliness 

0.0683 

-0.934 

0.0232 

1.06E+13 

0.0774 

1.37 

0.0331 

0.349 

0.0499 

Consume 

Essential 

Resources 

0.111 

1.589 

0.0140 

0.263 

0.0246 

0.595 

0.0434 

0.701 

0.0589 
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Linear 

Exponential 

Logarithmic 

Power 

Sigmoid  | 

WRMSE 

c 

WRMSE 

c 

WRMSE 

c 

WRMSE 

c 

WRMSE 

Consume 

Essential 

Time 

0.140 

-2.709 

0.0510 

9E+09 

0.140 

2.508 

0.0648 

0.528 

0.0612 

Friendly  at 
Risk  Cost 

0.224 

-0.808 

0.214 

310700 

0.224 

1.317 

0.210 

-0.437 

0.0588 

System 

Survivability 

Cost 

0.138 

-1.802 

1.38E-05 

1.93E+08 

0.138 

1.821 

0.0211 

-0.378 

0.684 

Collateral 

Damage 

Cost 

0.191 

1.11 

0.158458 

0.412 

0.162 

0.779 

0.170 

-0.437 

0.0588 

Sensitive 

Information 

Security 

Cost 

0.524 

1255 

0.15913 

13.293 

0.522 

0.098 

0.131 

-0.336 

0.0478 

Table  56.  RMSE  Model  Fit  Data  for  u (v (x)) . 


Linear 

Exponential 

Logarithmic 

Power 

Sigmoid  | 

RMSE 

c 

RMSE 

c 

RMSE 

c 

RMSE 

c 

RMSE 

Increase 

Recovery 

Time 

0.00470 

-0.045 

0.00332 

2.58E+08 

0.00470 

1.014 

0.00380 

0.528 

0.0494 

Degrade 

Update 

0.128 

0.516 

0.114 

1.628 

0.115 

0.871 

0.120 

0.292 

0.0337 

Increase 

Error 

Content 

0.196 

-3.745 

0.0728 

1.31E+1 1 

0.196 

3.396 

0.0853 

0.437 

0.0480 

Age 

Devaluation 

0.174 

-3.556 

0.0701 

8.08E+10 

0.174 

3.21 

0.0812 

0.529 

0.0494 

Decrease 

Accuracy 

0.155 

-0.962 

0.112 

2661000 

0.155 

1.421 

0.102 

0.336 

0.0399 

Reduce 

Timeliness 

0.0770 

-0.934 

0.0250 

1.06E+13 

0.0866 

1.37 

0.0349 

0.349 

0.0415 

Consume 

Essential 

Resources 

0.121 

1.589 

0.0143 

0.263 

0.0250 

0.595 

0.0402 

0.701 

0.0477 

Consume 

Essential 

Time 

0.148 

-2.709 

0.0540 

9E+09 

0.148 

2.508 

0.0663 

0.528 

0.0494 

Friendly  at 
Risk  Cost 

0.226 

-0.808 

0.206 

310700 

0.226 

1.317 

0.203 

-0.437 

0.0480 

System 

Survivability 

Cost 

0.156 

-1.802 

1.55E-05 

1.93E+08 

0.156 

1.821 

0.0220 

-0.378 

0.633 

Collateral 

Damage 

Cost 

0.191 

1.11 

0.163 

0.412 

0.168 

0.779 

0.171 

-0.437 

0.0480 

Sensitive 

Information 

Security 

Cost 

0.571 

1255 

0.178 

13.293 

0.569 

0.098 

0.113 

-0.336 

0.0399 
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Table  57.  Number  of  Best  Acceptable  Fits  By  Model  Type. 


RMSE 

WRMSE 

Linear 

0 

0 

Exponential 

5 

5 

Logarithmic 

0 

0 

Power 

0 

0 

Sigmoid 

7 

7 

None 

0 

0 

Table  58.  Total  Fits  by  Model  Type. 


RMSE 

WRMSE 

Number 

Percent 

Number 

Percent 

Linear 

2 

16.7 

4 

33.3 

Exponential 

9 

75.0 

9 

75.0 

Logarithmic 

4 

33.3 

4 

33.3 

Power 

10 

83.3 

9 

75.0 

Sigmoid 

11 

91.7 

11 

91.7 
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Abstract 

The  purpose  of  this  research  is  to  demonstrate  the  application  of  decision  analysis 
(DA)  techniques  to  the  decisions  made  throughout  the  lifecycle  of  Automatic  Target 
Recognition  (ATR)  technology  development.  This  work  is  accomplished  in  the  hopes  of 
improving  the  means  by  which  ATR  technologies  are  evaluated.  The  first  step  in  this 
research  was  to  create  a  flexible  decision  analysis  framework  that  could  be  applied  to  a 
variety  of  decisions  across  several  different  ATR  programs  evaluated  by  the 
Comprehensive  ATR  Scientific  Evaluation  (COMPASE)  Center  of  the  Air  Force 
Research  Laboratory  (AFRL).  For  the  purposes  of  this  research,  a  single  COMPASE; 
Center  representative  provided  the  value,  utility,  and  preference  functions  for  the  DA 
framework  through  elicitation  meetings  with  the  authors.  The  DA  framework  employs 
performance  measures  collected  during  ATR  classification  system  (CS)  testing  to 
calculate  value  and  utility  scores.  The  authors  gathered  data  from  the  Moving  and 
Stationary  Target  Acquisition  and  Recognition  (MSTAR)  program  to  demonstrate  how 
the  decision  framework  could  be  used  to  evaluate  three  different  ATR  CSs.  A  decision¬ 
maker  may  use  the  resultant  scores  to  gain  insight  into  any  of  the  decisions  that  occur 
throughout  the  lifecycle  of  ATR  technologies.  Additionally,  a  means  of  evaluating  ATR 
CS  self-assessment  ability  is  presented.  This  represents  a  new  criterion  that  emerged 
from  this  study,  and  no  present  evaluation  metric  is  known. 


n 


Table  of  Contents 


Title  Page . i 

Disclaimer . ii 

Abstract . iii 

Table  of  Contents . 44jv 

List  of  Figures  . . ....... . . . v 

List  of  Tables . vi 

1 .0  Summary . 1 

1.1  Introduction . 2 

2.0  Methods,  Assumptions,  and  Procedures  . . 3 

2. 1  The  Decision  Situation . 3 

2.2  Encoding  the  Value  Hierarchy . 4 

2.3  Single  Dimensional  Value  and  Utility  Function  Elicitation . 13 

2.4  ATR  CS  Alternatives . 15 

3.0  Results  and  Discussion . 16 

4.0  Conclusions . 22 

5.0  Recommendations . 23 

References . 24 

Appendix  A  ™  Elicitation  Results....... . 25 

List  of  Acronyms . 3 1 


[291] 


List  of  Figures 


Figure  2.1  Abstract  Depiction  of  Target  Truth  and  Clutter  Areas . 5 

Figure  2.2  Sample  Receiver  Operating  Characteristic  (ROC)  Curve . 8 

Figure  2.3  The  Value  Hierarchy  for  ATR  Decisions . 13 

Figure  3.1  Radar  Plots  of  ATR  Value  and  Utility . . IS 

Figure  3.2  Risk  Profile  for  CID . 19 

Figure  3.3  Risk  Profile  for  ISR . 19 

Figure  3.4  Sensitivity  Analysis  for  CID  Scenarios . 20 

Figure  3.5  Sensitivity  Analysis  foFISR  Scenarios . 20 

Figure  3.6  ATR  Value  (With  Certainty)  Versus  Redeployment  Cost . 21 

Figure  3.7  ATR  Value  (With.  Uncertainty)  Versus  Redeployment  Cost . 22 

Figure  3.8  ATR  Utility  (With  Uncertainty)  Versus  Redeployment  Cost . 22 


IV 


List  of  Tables 


Table  3.1  ATR  CS  Expected  Value  and  Expected  Utility  Results . 16 

Table  3.2  Recommended  ATR  CS  Alternative  by  Expected  Value  and  Utility ..  17 


[293] 


Application  of  Decision  Analysis  to  Automatic  Target  Recognition  Programmatic 

Decisions 

1.0  Summary.  The  purpose  of  this  research  is  to  demonstrate  the  implementation  of  a 
decision  analysis  (DA)  approach  towards  programmatic  decision-making  within  the 
Automatic  Target  Recognition  (ATR)  field  of  research.  Several  decisions  within  the 
lifecycle  of  ATR  research  are  based  upon  evaluations  using  many  performance  measures 
and  program  characteristics.  Often,  the  evaluation  of  these  raw  performance  measures 
and  characteristics  leads  to  non-dominating  solutions  among  the  various  ATR  programs 
and  technologies  competing  for  further  development.  Therefore,  the  decisions  are 
finalized  using  the  preferences  and  values  of  the  decision-maker.  Thus,  it  is  clear  that 
ATR  research  decision-makers  should  employ  a  formalized  decision  analysis  framework 
to  aid  in  selecting  the  best  ATR  research  direction  or  ATR  product  available  based  upon 
the  given  performance  characteristics  of  ATR  classification  systems  (CSs)  and  to  ensure 
evaluation  fairness  among  competing  technologies.  The  DA  framework  presented 
quantifies  and  rank-orders  the  preferences  of  the  decision-maker,  i.e.,  the  desirable  traits 
of  a  good  ATR  CS  are  weighted  more  heavily  than  less  desirable  traits.  The  numerous 
performance  measures  and  characteristics  of  a  given  set  of  ATR  CSs  may  then  be 
synthesized  through  the  DA  model.  The  result  is  a  single  utility,  or  value,  measure 
associated  with  each  ATR  CS  under  consideration.  This  methodology  provides  the 
decision-maker  with  a  defensible  approach  that  includes  his  own  value  structure  of  the 
problem  for  use  in  making  programmatic  decisions.  This  research  uses  past  ATR  data  to 
illustrate  that  this  technique  is  feasible  and  potentially  very  useful  to  the  decision-makers 
involved  in  ATR  research. 
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1.1  Introduction.  Automatic  Target  Recognition  is  a  processing  problem  where  an 
image  is  examined  in  order  to  detect  and  classify  objects  of  interest,  or  targets.  The 
image  is  provided  by  one  or  more  sensors,  which  typically  are  forward-looking  infrared, 
millimeter  wave,  synthetic  aperture  radar,  or  laser  radar  systems.  An  ATR  CS,  in  the 
form  of  a  pattern  recognition  and  classification  algorithm,  is  then  applied  to  the  image  in 
order  to  identify  particular  regions  of  interest  (ROIs)  and  to  then  classify  whether  the  ROI 
is  a  target  or  not.  It  is  necessary  to  point  out  that  both  enemy  and  friendly  objects  of 
interest  are  referred  to  as  targets.  Typically,  the  targets  are  difficult  to  separate  both  from 
normal  environmental  objects,  generally  referred  to  as  clutter ,  and  objects  with  target-like 
signatures  found  within  the  image,  often  referred  to  in  ATR  research  as  conjusers . 
Currently,  human  analysis  far  exceeds  the  capabilities  of  automated  ATR  systems.  It  is 
highly  desirable  to  improve  automated  ATR  capabilities,  which  would  increase  analytic 
capacity  in  military  intelligence  systems  as  well  as  permit  ATR  systems  to  be  employed 
on  unmanned  platforms.  Automatic  Target  Recognition  is  widely  acknowledged  as  a 
critical  military  capability  [4], 

ATR  CSs  generally  fall  into  three  classes:  statistical  pattern  recognition,  neural 
networks,  or  model-based  recognition.  All  ATRs  have  a  number  of  quantifiable 
evaluation  measures,  such  as  ATR  CS  performance,  robustness,  estimate  accuracy, 
employment  doctrine,  and  cost,  which  may  be  used  to  compare  multiple  ATR  CSs.  In 
general,  these  measures  are  not  assessed  in  total,  but  specific  measures  are  selected  when 
considering  decisions  for  a  specific  program.  No  examples  of  employment  of  DA 
techniques  with  respect  to  ATR  selection  have  been  found  in  the  literature. 
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2.0  Methods,  Assumptions,  and  Procedures.  The  following  section  details  the  steps 
taken  to  construct  a  DA  framework.  The  first  step  j.s  defining  the  decision  situation  with 
a  decision-maker.  Next,  the  DA  model  is  generated  using  quantitative  representations  of 
the  preferences  held  by  the  decision-maker.  Finally,  the  performance  measures  of 
multiple  ATR  CSs  may  then  be  introduced  to  the  model  for  analysis. 

2.1  The  Decision  Situation.  The  objective  of  ATR  evaluation  is  to  determine  which 
system  performs  best  via  performance  measure  assessment  and  comparison.  During 
development  of  past  ATR  CSs,  decisions  that  compared  competing  ATR  CSs  were  not 
performed  in  a  manner  that  collectively  considered  all  pertinent  aspects  of  the  decision. 
Typically  a  subset  of  criteria  was  examined.  Employment  of  decision  analytic  techniques 
permits  all  criteria  to  he  considered  and  trade-offs  performed  when  comparing  systems . 
As  an  example  of  how  these  techniques  can  be  applied,  the  Moving  and  Stationary  Target 
Acquisition  and  Recognition  (MSTAR)  program,  which  was  an  investigative  effort  into 
synthetic  aperture  radar  (SAR),  model-based  ATR  development,  provides  an  ideal 
scenario  [5].  At  the  time,  three  ATR  systems  were  tested  as  candidates  for  furthering  the 
development  of  an  advanced  ATR  system  [5].  These  ATR  CSs  are  referred  to  as  ATR 
33,  ATR  55,  and  ATR  89. 

There  are  two  basic  ATR  employment  profiles  under  consideration.  The  Combat 
Identification  (CID)  employment  profile  is  implemented  when  the  primary  objective  of 
the  ATR  CS  is  to  select  targets  for  weapon  systems.  In  this  scenario,  the  system  is 
allowed  to  sacrifice  detection  performance  in  order  to  gain  classification  accuracy.  Thus, 
the  selection  of  a  target  must  be  associated  with  a  high  degree  of  confidence  as  to 
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minimize  the  number  of  false  alarms.  The  Intelligence/S urveillance/Reconnaissance 
(ISR)  employment  profile  is  used  when  the  primary  objective  of  the  ATR  CS  is  to  collect 
information  for  many  potential  targets.  In  this  scenario,  the  CS  is  allowed  to  sacrifice 
classification  accuracy  for  improved  detection  performance.  Thus,  the  goal  of  the  ATR 
CS  is  to  detect  as  many  potential  targets  as  possible  as  to  minimize  the  number  of  targets 
that  evade  detection. 

In  order  to  assist  the  Comprehensive  ATR  Scientific  Evaluation  (COMPASE) 
Center,  an  Air  Force  Research  Laboratory  (AFRL)  organization  within  the  Sensors 
Directorate,  with  its  assessment  of  which  future  ATR  systems  should  be  retained  for 

^  J 

further  development,  a  feasibility  study  of  value-focused  decision  analysis  is  performed 
upon  the  MSTAR  data.  Dr.  Timothy  Ross,  COMPASE  Center  Director,  served  as  the 
decision-maker.  Dr.  Ross  participated  in  two  elicitation  sessions.  The  initial  session 
developed  the  subject’s  value  hierarchy  and  elicited  single  dimensional  value  functions 
for  the  evaluation  measures.  Single  dimensional  utility  functions  were  elicited  during  the 
second  session  as  well  as  follow  up  clarification  on  several  minor  issues.  Additionally, 
the  authors  discussed  minor  points  with  Dr.  Ross  via  e-mail.  Throughout  the  elicitation 
process,  the  subject  was  comfortable  expressing  his  preferences  in  a  quantified  manner. 
Discussions  with  Dr.  Ross  confirmed  and  refined  the  evaluation  measures  for  the 
decision  situation. 

2.2  Encoding  the  Value  Hierarchy.  In  a  test  environment,  ATR  CS  evaluation  is 
accomplished  using  performance  measure  assessment  and  comparison.  Assessment 
refers  to  the  collection  of  several  quantitative  performance  measures.  Understanding  the 
design  of  the  testing  environment  is  key  to  the  understanding  of  performance  measure 
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assessment.  To  begin,  the  test  image  is  broken  into  two  separate  areas:  the  target  truth 
area  and  the  clutter  scene  area.  Testing  personnel  place  targets  and  confusers  within  the 
target  truth  area.  The  ATR  CS  examines  this  section  of  the  scene  for  potential  targets. 
Any  target  declaration  made  against  a  confuser,  or  non-targets,  is  considered  a  false 
alarm.  Next,  the  ATR  CS  scans  the  clutter  scene  area,  which  is  known  to  be  devoid  of 
targets.  Any  target  declarations  made  by  the  ATR  CS  in  this  section  are  considered  to  be 
false  alarms  in  clutter,  since  no  confusers  exist  in  this  area.  Figure  2.1  illustrates  the 
manner  in  which  an  ATR  test  area  is  divided.  When  an  ATR  CS  has  scanned  both  areas, 
ATR  performance  may  then  be  assessed. 


Target  Truth  Area 

Clutter  Area 

p  a  - ' : 

Blf 

sHw™ 

"A: 

■  "  I  C  H 

Key:  T  ->  Target  N  ->  Confuser 

C  Clutter  □  Declared  as  Target 

Figure  2-1  Abstract  Depiction  of  Target  Truth  and  Clutter  Areas. 


The  primary  performance  measure  is  the  probability  of  detection,  PD.  This 
measure  is  defined  as: 

P  n  -  Nqet  !  N tgt  ( 1 ) 
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where  NDET  is  the  number  of  targets  detected  by  the  ATR  CS  and  Ntgt  is  the  total  number 
of  targets  to  be  detected  within  the  image.  The  probability  of  detection  provides  insight 
into  how  well  the  ATR  CS  is  detecting  the  target  it  is  designed  to  find.  Beyond  detecting 
targets,  it  is  desirable  that  the  ATR  CS  provides  a  conjecture  of  the  taxon,  or  label,  of  the 
object.  The  ATR  may  or  may  not  further  refine  the  description  of  the  target.  Typically,  a 
refined  identification  is  bifurcated  into  class  and  type,  but  a  target  may  be  labeled  at  any 
level  desirable  to  the  decision-maker.  Class  describes  a  broad  category  of  materiel. 
Example  class  taxa  are  main  battle  tanks  (MBTs)  and  surface-to-air  systems.  The  type  of 
the  object  is  the  specific  nomenclature.  The  M-1A1  and  T-72  main  battle  tanks  are  ‘type’ 
examples  of  tbt  MBT  class.  Given  the  targets  detected  by  the  ATR  CS,  additional 
performance  measures  conditional  to  PD  may  then  be  calculated  concerning  how  well  the 
ATR  algorithm  classifies  the  ROls  that  it  considers  targets.  The  first  performance 
measure,  the  probability  of  correct  classification  (Pec),  is  the  ratio  of  the  number  of 
targets  correctly  classified  by  class  (Ncc)  to  the  total  number  of  detected  targets  ( AW )■ 
Thus,  Pec  is  defined  as: 

Pee  =  NcdNbET-  *  (2) 

In  other  words,  Pec  measures  the  proportion  of  correctly  labeled  targets  to  the  number  of 
correctly  detected  targets,  e.g.,  classifying  a  detected  target  as  a  MBT  when  it  indeed  is  a 
MBT,  and  serves  as  an  estimate  of  the  probability  of  correctly  classifying  a  target.  A 
similar,  but  slightly  more  specific,  performance  measure  is  the  probability  of  correct 
identification,  or  P id-  This  measure  is  the  ratio  of  the  number  of  targets  correctly 
classified  by  type  (Nm)  to  the  total  number  of  detected  targets  (AW).  Thus,  Pm  is 
defined  as: 
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Pid-Nid/Ndet ■  (3) 

In  other  words,  PID  measures  the  proportion  of  correctly  labeling,  for  example,  a  ROI  as  a 
T-72  MBT  when  it  indeed  is  a  T-72  MBT. 

While  the  previous  performance  measure  indicate  how  well  an  ATR  CS  performs 
in  detecting  targets,  the  next  set  of  measures  provides  insight  into  how  often  an  ATR  CS 
mistakes  non-targets  as  targets.  The  probability  of  false  alarm  measure,  denoted  PFA>  is 
the  ratio  of  the  number  of  detected  confusers,  or  non-targets,  (Nc)  to  the  total  number  of 
known  confusers  in  the  image  (Nfa),  and  is  defined  as: 

P fa  =NC  I  Nfa.  (4) 

A  similar  performance  measure  is  the  false  alarm  rate  (FAR),  which  is  the  ratio  of  the 
number  of  false  alarms  in  clutter  (NCL)  to  the  clutter  scene  area  (A),  defined  as: 

FAR  -  Net  /  A.  (5) 

This  performance  measure  indicates  how  likely  an  ATR  is  to  mistake  terrain  or  natural 
objects  as  a  potential  target. 

Typically,  Pd  is  changed  for  an  ATR  algorithm  by  adjusting  a  detection  threshold 
internal  to  the  CS.  PD  may  be  increased  to  any  desired  level  as  the  detection  threshold  is 
adjusted,  but  there  is  a  corresponding  degradation  in  the  ATR  performance  as  more 
clutter  and  confusers  are  incorrectly  declared  as  targets.  That  is,  the  false  alarm  rate 
(FAR)  and  probability  of  false  alarm,  PM,  never  decrease,  and  should  increase,  as  the 
probability  of  detection  increases.  For  a  given  ATR  CS,  a  receiver  operating 
characteristic  (ROC)  curve  illustrates  the  trade-off  between  the  detection  performance 
and  the  false  alarm  rate.  Figure  2.2  depicts  sample  ROC  curves  along  with  an  area  under 
the  curve  measure,  which  is  typically  used  in  comparing  multiple  ROC  curves.  When 
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evaluating  ATRs,  either  the  P0  is  fixed  for  a  particular  mission  profile,  or  the  ROC  curve 
is  employed.  In  both  cases,  Pd  is  not  considered  an  evaluation  measure  as  the  ATR 
performs  at  that  level  by  definition.  The  false  alarm  performance  is  then  the  concern. 


Probability  of  F alse  Alarm 

Figure  2.2  Sample  Receiver  Operating  Characteristic  (ROC)  Curve  [2],  The 

ROC  curve  represents  the  performance  of  a  given  ATR  CS  as  an  internal  detection 
threshold  is  varied.  The  plot  above  illustrates  the  ROC  curve  for  four  different  ATR 
CSs  and  provides  a  sample  area  under  the  curve  (A)  performance  measure  for  each 
(higher  is  better). 


Another  desirable  characteristic  of  an  ideal  ATR  CS  is  its  frequency  of  making 
declarations  of  targets  and.  non-targets  to  be  detected.  Thus,  a  superior  ATR  system  will, 
declare  a  larger  set  of  the  known  target  population,  in  an  image  than  an  inferior  system. 
This  measure  hints  at  the  confidence  an  ATR  has  in  its  detection  ability.  The  probability 
of  declaration,  or  Pdec,  is  defined  as: 
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P  dec  =  Ndec  /  N total  (6) 

where  NDBc  represents  the  number  of  correct  declarations  (declaring  a  ROI  to  either  a 
target  or  a  non-target)  made  by  the  ATR  CS  and  Ntotal  is  the  total  number  of  test  objects, 
both  targets  and  confusers.  Thus,  Ndec  consists  of  all  ROIs  declared  as  targets  that  are 
targets  and  all  ROIs  declared  to  be  non-targets  by  the  ATR  CS  that  are  confusers. 

It  is  also  desirable  that.  ATR  performance  is  robust  to  target,  environmental,  and 
sensor  differences;  provide  an  assessment  of  its  confidence  in  its  target  estimates;  and 
have  a  well-developed  employment  concept.  ATR  CSs  are  typically  trained  against  a 
given  set  of  baseline,  or  nominal ,  target  images.  An  example  of  ATR  performance 
robustness  occurs  when  an  ATR  C.S  is  able  to  detect  and  possibly  correctly  classify  an 
MBT  even  though  it  has  not  been  presented  with  the  particular  configuration  that  the 
vehicle  exhibits.  Thus,  the  ATR  CS  detects  and/or  correctly  classifies  the  MBT  when  a 
variety  of  external  differences  from  the  nominal  image  training  set,  such  as  open  hatches, 
turret  articulation,  or  external  stores,  appear.  Performance  measures  based  upon  the 
nominal  training  set,  which  are  generally  near  perfect,  are  then  compared  to  the 
performance  measures  where  at  least  one  change  is  made  in  the  target  configuration.  It.  is 
desirable  that  the  probabilities  of  detection,  typification,  and  classification  (PD,  Pm,  Pcc) 
remain  as  close  to  their  nominal  values  as  possible.  Degradation  is  measured  in  percent 
change  from,  the  nominal  values  for  some  specific  target  set  where  the  targets  are 
perturbed  in  some  fashion.  The  percent  change  values  for  probability  of  detection  are 
calculated  in  the  following  manner,  and  PCc  and  Pro  are  assessed  similarly.  A  nominal 
probability  of  detection  measure  ( Pd.Nom )  is  assessed  against  targets  at  the  baseline 
configuration.  Next,  a  probability  of  detection  measure  is  assessed  against  all  other  target 
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configuration  deviations  of  interest  (, Pd-dev ),  which  may  include  sensor,  target, 
environmental,  or  some  combination  of  changes  to  the  configuration.  The  difference 
between  the  two  measures  is  assessed: 


=  Pd-nom  -  Pd-dev  (7) 

and  the  resultant  difference  estimate  is  between  0  and  1,  For  a  large  number  of 
observations  on  a  target  array,  the  Central  Limit  Theorem  permits  the  assumption  of 
normality,  and  the  formula: 


$(X)=A PD±Zff, 


APd(1-APd) 

n 


(8) 


can  be  used  to  create  a  probability  density  function,  @(x),  around  the  estimate  for  use  in 
the  decision  analysis  model  [3].  To  complete  the  procedure  and  create  a  percent  change 
in  the  detection  difference  measure  (%APp),  the  upper,  lower,  and  mean  values  of  the 
probability  of  detection  difference  estimate  must  be  multiplied  by  100.  Though  these 
estimates  could  be  expressed  as  a  normal  distribution  within  the  decision  analysis  model, 
they  may  be  approximated  by  a  triangular  distribution.  The  triangular  distribution  forces 
the  realizations  to  be  bounded  within  the  domain  of  elicitation  of  the  DA  information. 

The  ATR  CS  self-assessment  is  a  confidence,  expressed  as  a  probability  of 


accuracy,  for  the  detection,  typification,  and  classification  estimates,  CD ,CID ,Ccc , 
respectively,  as  determined  by  the  CS.  For  example,  an  ATR  CS  may  have  difficulty 
with  the  correct  identification  of  a  target.  Perhaps  the  target  exhibits  characteristics  that 
indicate  to  some  degree  that  it  is  a  T-72  MBT,  but  other  characteristics  indicate  that  it  is  a 
T-80  MBT.  The  ATR  CS  provides  an  identification,  declaring  that  it,  is  either  a  T-72  or  a 
T-80,  and  an  associated  confidence,  Clp ,  for  its  declaration.  A  CS  may  be  designed  to 
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return  a  single  target  type  with  an  associated  confidence  (e.g.,  90  percent  confidence  that 
the  target  is  a  T-80),  or  a  list,  of  several  possible  target  types  each  with  an  associated 
confidence  (e.g,,  60  percent  confidence  that  the  target  is  a  T-80  and  40  percent 
confidence  that  the  target  is  a  T-72).  Detection  and  classification  confidences  perform 
similarly. 

This  self-confidence  score  is  compared  to  the  true  target  identity  to  assess  the 
accuracy  of  the  confidence  estimate.  The  accuracy  of  these  confidence  estimates  for 
detection,  typification,  and  classification  estimates  (ES-pp,  Es.pcc,  ES-pid )  are  scored  on  the 
unit  interval.  The  inclusion  of  the  self-assessment  measure  accuracy  in  the  DA  model  is 
for  use  on  future  ATR  systems  with  this  capability.  The  ATR  CSs  of  the  MSTAR 
program  did  not  have  the  capability  to  provide  or  use  these  measures. 

One  possible  method  of  assessing  ES-pd,  Es.PCc ,  Es-pw  would  be  to  compare 
C°  ,CID ,Ccc  to  the  true  nature  of  the  target.  Continuing  with  the  identification  case, 
each  target  is  assigned  C'f  by  the  ATR  CS,  where  the  variable  i  indicates  the  target  and 


the  variable  j  indicates  various  identifications  of  the  same  target,  i  by  the  CS.  The  self- 
assessment  accuracy,  Bs  pid,  may  be  determined  from 
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where  J  is  the  total  number  of  possible  target  identifications  for  the  ith  target,  N  is  the 
total  number  of  targets,  M  is  the  total  number  of  target  identifications  made  by  the  CS, 


and  T  is  an  indicator  variable  defined  as: 
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ID  correct 
ID  incorrect 


Similarly,  ES-pd,  and  ES.PCC  are  defined  by 


and 


respectively. 


1  IV  J 

E,.m  =-ZX 

iv  1=1  j=\ 


i-« — - - 


LV 


t(cf) 


1  N  J 

Es-pcc  =  T7  XX 

■/V  i=l  j?  1 


1-J a - r- - - - 


LV 


iCC 

A 

1=1  7 


MO 


(10) 


(11) 


(12) 


Finally,  cost  is  a  consideration.  Costs  may  be  grouped  by  developmental  costs, 
redeployment  costs,  and  operational,  or  use,  costs.  Developmental,  costs  are  those 
incurred  when  bringing  the  ATR  to  an  operational  status.  Redeployment  costs  are  those 
incurred  when  moving  the  ATR  from  its  base  to  an  operational  area.  Operational  costs 
are  those  incurred  when  employing  the  system.  Each  of  these  categorical  costs  may  be 
further  decomposed  into  the  sub-costs  of  funding,  time,  expertise,  and  risk,  except  that 
there  are  no  operational  risks  (with  respect  to  the  system  performance). 

The  value  hierarchy  that  emerged  from  the  elicitation  is  depicted  in  Figure  2.3. 
The  parenthetical  numbers  are  the  weights  representing  the  relative  importance,  within 
the.  parent  value,  of  the  values.  The  value  hierarchy  was  decomposed  until  the  bottom 
tier  members  were  measurable. 
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Influence  ATI?  Technology  Decision 
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Figure  2.3  The  Value  Hierarchy  for  ATR  Decisions.  The  dashed  line  under 
Overall  Detection  Performance  indicates  that  either  the  Defined  PD  or  the  ROC 
measures  are  employed.  The  parenthetical  numbers  indicate  the  relative  weights  for  a 
value,  within  the  parent  value. 


In  order  to  assess  the  categorical  data,  scales  were  constructed.  For  the  adequacy 
of  employment  concept  measure,  the  created  scale  categorized  the  employment  concept 
as  well  defined,  strongly  defined,  moderately  defined,  poorly  defined,  and  no  definition. 
Cost  risks  were  assessed  on  a  scale  with  low,  medium,  and  high  risk.  Because  these 
scales  were  developed  in  cooperation  with  the  subject,  no  precise  definitions  were 
specified  for  the  terms. 


2.3  Single  Dimensional  Value  and  Utility  Function  Elicitation.  The  single 
dimensional  value  functions  were  elicited  for  each  of  the  evaluation  measures  (the  bottom 
tier  of  the  value  hierarchy).  It  was  explained  that  roost  and  least  preferred  levels  of  an 
evaluation  measure  mapped  to  one  and  zero  respectively,  and  the  function  captures 
intervening  preference  levels.  Initial  discussion  introduced  the  idea  of  comparing  relative 
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differences  in  preference  between  pairs  of  evaluation  measure  observations.  However, 
the  subject  was  very  comfortable  with  expressing  preferences  in  a  quantitative  manner, 
usually  commenting  on  a  functional  form  and  providing  a  value  to  fix  the  curve.  For 
example,  he  often  observed  that  his  preference  would  decrease  exponentially  on  a 
measure  and  pass  through  a  point  he  specified.  Direct  assessment  of  the  value  functions 
was  employed. 

Single  dimensional  utility  functions  were  elicited  for  continuous  evaluation 
measures  employing  the  certainty  equivalent  lottery  technique.  In  this  process  the  subject 
chooses  between  uncertain  alternatives  with  two  equally  likely  outcomes,  each  at  the 
extremes  of  the' evaluation  measure  domain,  or  a  certain  alternative  with  some  specified 
level  of  the  evaluation  measure.  The  certain  alternative  is  varied  until  the  subject  is 
indifferent  between  the  alternatives.  As  is  considered  standard  practice,  the  certainty 
equivalent  was  approached  'by  alternately  providing  values  from  the  margins  of  the 
evaluation  measure  domain  to  avoid  anchoring.  The  initial  lottery  considered  the  entire 
domain  for  the  evaluation  measure  under  consideration  at  that  point.  This  provides  the 
datum  point  (known  as  a  mid-value  point)  for  which  the  subject  assigns  a  utility  of  0.5. 
This  process  was  repeated  for  the  sub-domains  created  about  the  mid-value  point  (the 
fractile  approach).  Thus  each  utility  function  is  established  with  five  data  points,  three  of 
which  are  elicited  and  two  defined.  The  subject  again  was  comfortable  with  the 
elicitation  methodology  and  often  quickly  determined  his  indifference  points.  As  no 
functional  form  was  evident  from  these  elicitations,  linear  interpolation  was  employed  for 
utilities  between  elicited  points. 
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For  the  evaluation  measures  with  categorical  scales,  the  probability  equivalent 
lottery  method  was  used.  The  subject  is  provided  a  choice  between  uncertain  and  certain 
alternatives.  The  uncertain  alternative  provides  the  most  preferred  level  of  the  alternative 
with  some  probability  p  and  the  least  preferred  with  probability  (1-p).  The  certain 
alternative  provides  the  evaluation  measure  level  for  which  the  corresponding  utility  will 
be  determined.  The  p  is  varied  until  the  subject  is  indifferent.  The  utility  of  the  certain 
alternative  is  then  equal  to  p.  By  considering  each  category  between  the  least  and  most 
preferred  levels,  the  utility  for  each  category  is  determined.  Again  the  subject  rapidly 
provided  indifference  points  for  p. 


2.4  ATR  CS  Alternatives.  •  Data  for  most  of  the  evaluation  measures  of  the  three 
alternatives,  ATR  33,  ATR  55,  and  ATR  89,  were  available  from  past  MSTAR  test  data. 
As  the  MSTAR  data  did  not  include  all  evaluation  measures,  assumptions  were  made  to 
permit  analysis.  These  assumptions  were: 

•  Developmental  Costs  are  sunk.  Value,  v(x),  and  utility,  u(x)  are  set  equal 
to  one  in  the  model. 

•  Robustness  for  classify  by  class  (%  A  PCc)  data  was  not  collected.  Data 
have  v(x)  =  u(x )  =  1. 

•  ATR  CSs  evaluated  in  the  MSTAR  program  did  not  have  the  self- 
assessment  capability:  v(x)  =  «(x)  =  0. 

•  No  Classify  by  Class  data  was  collected,  so  it  was  assumed  that  Pec  =  Pid- 
This  provides  a  conservative  estimate  of  Pcc. 

•  Assumed  that,  costs  for  use  time  greater  than  300  CPU  seconds  have 
v(x)=«(x)  =  0and  were  truncated  to  stay  within  elicited  domain  for  this 
evaluation  measure. 

•  The  preference  functions  for  the  cost  of  redeployment  of  the  system  were 
elicited  on  a  domain  normalized  for  the  Global  Hawk  system.  The  Global 
Hawk  costs  were  assumed  to  be  $60k  for  this  study. 

•  Operational  costs  were  based  on  a  $10k  price  for  a  workstation  with  a 
three-year  lifecycle. 

•  Distributions  were  assumed  to  be  triangular. 
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•  Pdec  for  all  ATR  CSs  is  assumed  to  be  1  since  MSTAR  CSs  must  make  a 
declaration  decision. 

Some  data  were  fictionalized  from  their  test  values  to  mask  competition  sensitive 
information.  Fictional  data  are  representative  and  the  results  may  be  used  to  validate  the 
methodological  approach,  but  the  results  are  not  valid  for  ATR  selection.  The  analysis 
would  have  to  be  repeated  with  true  data  for  an  ATR  selection  decision.  The  data  for  the 
alternatives  is  available  through  the  COMP  AS  E  Center. 


3.0  Results  and  Discussion.  The  expected  value  results  for  the  alternatives  examined 
with  value  functions  (under  conditions  of  uncertainty  and  certainty  where  the  mean  was 
treated  as  deterministic)  and  utility  functions  (under  uncertainty)  are  provided  in  Table 
3.1.  The  results  for  the  value  functions  under  certainty  differ  slightly  from  those 
employing  the  value  function  under  uncertainty  (note  the  CID  profile  for  ATR  55).  This 
illustrates  that  using  the  mean  of  a  distribution  and  treating  it  deterministically  and  using 
a  value  function  does  not  provide  identical  results  as  considering  the  stochastic  nature  of 
the  problem  within  the  analysis. 


Table  3.1  ATR  CS  Expected  Value  and  Expected  Utility  Results. 


ATR  33 

ATR  55 

ATR  89 

Value 

Functions 

(Certainty) 

CID 

0.509 

0.537 

0.525 

ISR 

0.497 

0.531 

0.497 

Value 

Functions 

(Uncertainty) 

CID 

0.509 

0.556 

0.525 

ISR 

0.497 

0.531 

0.497 

Utility 

Functions 

(Uncertainty) 

CID 

0.572 

0.507 

0.518 

ISR 

0.414 

0.455 

0.439 
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Comparing,  by  rank  ordering  the  alternatives,  the  results  of  the  value  functions 
under  uncertainty  and  the  utility  functions  in  Table  3.2,  we  see  that  the  recommended 
alternative  differs  for  the  CID  case.  This  indicates  that  the  constructs  of  value  and  utility 
provide  differing  answers.  As  the  problem  involves  uncertainty,  the  utility  results  are  the 
appropriate  choice.  The  two  employment  profiles  provide  differing  recommendations. 
For  CID  missions,  ATR  33  is  the  best  choice.  For  ISR  missions,  ATR  55  is  the  preferred 
alternative.  However  ATR  89  was  the  second  choice  under  both  profiles.  Should  one 
ATR  be  required  to  perform  both  missions,  ATR  89  may  be  the  most  appropriate  choice. 
These  results  are  depicted  graphically  in  Figure  3.1.  The  risk  profiles  (as  cumulative 
distribution  functions  [CDFs]  are  referred  to  in  DA  terminology)  for  the  utility 
distributions  for  the  CID  and  ISR  profiles  are  provided  in  Figures  3.2  and  3.3.  Figures 
3.4  and  3.5  present  tornado  diagrams  for  the  two  profiles  where  the  weights  were  varied 
by  ten  percent.  The  lack  of  color  change  (shading)  in  the  bars  indicates  that  the  ATR 
choice  is  not  sensitive  to  the  weights  elicited  from  the  decision  maker. 


Table  3.2  Recommended  ATR  CS  Alternative  by  Expected  Value  and  Utility. 


ATR  33 

ATR  55 

ATR  89 

Value  Functions 
(Certainty) 

CID 

3 

1 

2 

ISR 

2 

1 

2 

Value  Functions 
(Uncertainty) 

CID 

3 

1 

2 

ISR 

2 

1 

2 

Utility  Functions 
(Uncertainty) 

CID 

1 

3 

2 

ISR 

3 

1 

2 
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CID 


ATR 


ATR33 


ISR 


ATR  33 


Figure  3.1  Radar  Plots  of  ATR  Value  and  Utility.  Value  results  are  represented  by 
solid  lines,  utility  by  dashed.  The  Combat  Identification  (CID)  results  are  above,  the 
Intelligence/Surveillance/Reconnaissance  (ISR.)  are  below. 
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Figure  3.3  Risk  Profile  for  ISR. 
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Figure  3.4  Sensitivity  Analysis  for  CID  Scenarios. 
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Figure  3.5  Sensitivity  Analysis  for  ISR  Scenarios. 
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The  following  figures  illustrate  that  differing  approaches,  deterministic  value, 
uncertain  value  and  uncertain  utility,  provide  differing  answers.  For  the  certain  value 
approach,  Figure  3.6  shows  that  for  the  CID  employment  profile,  increasing  cost  of  the 
ATR  CS  provides  increasing  value  (benefit).  For  the  ISR  profile,  ATR  33  dominates 
ATR  89.  ATR  55  does  provide  improved  value  for  an  increased  cost.  These  results  hold 
for  the  uncertain  value  case,  as  illustrated  in  Figure  3.7.  When  utility  is  employed  for  the 
CID  profile  (Figure  3.8),  ATR  33  dominates  ATR  89,  which  in  turn  dominates  ATR  55. 
For  the  ISR  profile,  increasing  cost  provides  increasing  utility  (benefit).  These 
interpretations  agree  with  those  of  Table  3.2.  Clearly  both  the  mission  profile  and  the 
DA  methodology  significantly  affect  the  recommendation  (answer). 


Figure  3.6  ATR  Value  (With  Uncertainty)  Versus  Redeployment  Cost. 
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Figure  3.7  ATR  Value  (With  Uncertainty)  Versus  Redeployment  Cost. 


♦  CID 
■  ISR 


AIR  55 


AIR  89 


ATR  33 


0.58 
0.56 
0.54 
0.52 
£  0.5 
§  0.48 
0.46 
0.44 
0.42 
0.4 


Figure  3.8  ATR  Utility  (With  Uncertainty)  Versus  Redeployment  Cost. 
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4.0  Conclusions.  The  conclusion  can  be  drawn  that  the  DA  approach  does  indeed  work. 
The  decision-maker’s  preferences  are  encapsulated  in  the  framework,  and  the  resultant 
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utility  measure  provides  a  defensible  argument  towards  selecting  a  given  ATR  CS  over 
another. 

5.0  Recommendations.  The  authors  recommend  that  these  results  should  be  applied 
towards  a  warfighter’s  perspective  of  the  same  problem.  In  a  method  parallel  to  the  one 
detailed 'in  this  report,  a  DA  framework  ‘  should  be  constructed  that  provides  a  utility 
function  measure  for  combat  model  results  generated  using  similar  ATR  CS  performance 
measures.  The  utility  of  the  warfighter  perspective  may  then  be  used  in  conjunction  with 
the  ATR  technology  developer’s  utility  results  in  order  to  positively  affect  the  manner  in 
which  ATR  products  are  evaluated  and  judged. 

It  is  recommended  that  the  decision  analysis  approach  should  be  incorporated  into 
ATR  research  and  development  programs.  Design  of  experiments  and  data  collection 
efforts  should  serve  to  provide  data  to  score  alternatives  in  accordance  with  the  value 
hierarchy.  Assessment  of  ATR  programs  should  include  mission  profile  considerations. 
Decision  analysis  evaluations  should  be  utility  based  unless  all  criteria  are  considered  to 
be  under  conditions  of  certainty,  which  is  unlikely. 
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Appendix  A  -  Elicitation  Results 

The  results  of  the  elicitation  meetings  between  the  authors  and  the  COMPASE 
Center  representative  are  summarized  below.  Elicitation  sessions  also  included  hand- 
drawn  plots  of  the  value  functions,  which  are  not  presented  here. 


1 .  Robustness. 

a.  Top  level  weight:  0.2 

b.  Subvalues: 

i.  Detection  Robustness 

1.  Units:,  percent  change  in  Pd  , 

2.  Domain:  [o,l]. 

3.  Weight:  0.425. 

4.  Value  Function:  exponential  fitting  {(o,l)  (25,0,2)  (50,0)). 

5.  Utility  Function:  {(0,l)  (3,0.75) (l0,0.5Kl0, 0.25)  (50,0)}. 
.ii.  Identification  Robustness 

1.  Units:  percent  change  in  Pid . 

2.  .Domain:  [o,l], 

3.  Weight:  0.275. 

4.  Value  Function:  exponential  fitting  {(0,l)  (25,0.2)  (50,0)}. 

5.  Utility  Function:  {(0,l)  (3,0.75) (l 0,0.5)  (l 0,0.25)  (50, 0)j. 
iii.  Classification  Robustness 

1.  Units:  percent  change  in  Prr . 

2.  Domain:  [0,l]. 

3.  Weight:  0.3. 

4.  Value  Function:  exponential  fitting  {(0,l)  (25,0.2)  (50,0)}. 

5.  Utility  Function:  {(0,l)  (3,0.75)  (10,0.5)  (l 0,0.25)  (50, 0)). 

2.  Overall  Detection  Performance.  Note  that  this  value  decomposes  into  either  the 
“defined  Pd  ”  subvalue  or  the  “ROC”  subvalue  set,  depending  on  which  metric  is 
used. 

a.  Top  level  weight:  0. 17. 

b.  Subvalues: 

i.  Defined  Pd . 

1 .  FAR  I  Pd  .  Note,  FAR  I  Pd  =  0.9  or  FAR  I  Pd  =  0.5  , 
depending  on  mission  profile. 

a.  Units:  occurences  / km 2 . 

b.  Domain:  [0,l]. 
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c.  Weight:  0.429. 

d.  Value  Function:  exponential  fitting 
{(0,l)(l,0.3)  (1000,0)} 

e.  Utility  Function: 

{(0,1)  (3,0.75)  (10,0.5)  (50,0.25)  (l  000,0)}. 

2-  p fa  I  P,t  ■  Note,  PFA  I  Pj  -  0,9  or  PFA  I  Pd  =  0.5 ,  depending 
on  mission  profile. 

a.  Units:  probability. 

b.  Domain:  [0,l]. 

c.  Weight:  0.571. 

d.  Value  Function:  linear  fitting  {(0,l)  (l,o)}. 

e.  Utility  Function: 

{(0,1)  (0. 1,0.75)  (0.3,05)  (0.4,0.25)  (l,0)}. 

ii.  ROC. 

1.  ROC  FAR. 

a.  Units:  area  under  ROC  curve. 

b.  Domain:  [0,1000]. 

c.  Weight:  0.429. 

d.  Value  Function:  exponential  fitting 
{(0,0)  (500,0.1)  (1000,1)}. 

e.  Utility  Function: 

{(0,0)  (600,0.25)  (750,0.5)  (875,0.75)  (l  000,1)}. 

2.  ROC  PFA . 

a.  Units:  normalized  area  under  ROC  curve. 

b.  Domain:  [0,l], 

c.  Weight:  0.571. 

d.  Value  Function:  exponential  fitting 
{(0.5,0)  (0.75,0.3)  (1,1)}. 

e.  Utility  Function: 

{(0.5,0)  (0.8,0.25)  (0.875,0.5)  (0.938,0.75)  (l,l)}, 

3,  Employment  Concept. 

a.  Units:  categorical. 

b.  Domain:  {None,  Poorly  Defined,  Moderately  Defined,  Strongly  Defined, 
Well  Defined}. 

c.  Weight:  0.15. 

d.  Value  Function:  { (None,  0),  (Poorly  Defined,  0,4),  (Moderately  Defined, 
0.6),  (Strongly  Defined,  0.9),  (Well  Defined,  1)}. 

e.  Utility  Function:  { (None,  0),  (Poorly  Defined,  0.1),  (Moderately  Defined, 
0.5),  (Strongly  Defined,  0.9),  (Well  Defined,  1)}. 

4,  Declaration  Ability. 

a.  Units:  probability. 

b.  Domain:  [o.5,l]. 

c.  Weight:  0.13. 


26 


CF"  CJ 


[319] 


d.  Value  Function:  exponential  fitting  {(0,0)  ((). 5,0,7)  (l,l)). 

e.  Utility  Function:  {(0,0)  (0.25,0.25) (0.5.0.5)  (0.75,0.75)  (ltl». 

5.  Classification  Ability. 

a.  Top  level  weight:  0. 1 1 . 

b.  Subvalues: 

i.  Correctly  Classify  by  Type  (P/p  ) . 

1.  Units:  probability. 

2.  Domain:  [0.5,l]. 

3.  Weight:  0,474. 

4.  Value  Function:  exponential  fitting  {(0,5,0)  (0.75,0.3)  (l,l)}. 

5.  Utility  Function: 

{(0.5,0)  (0.6,0.25)  (0.7, 0.5)  (0.9,0.75)  (l,l)}. 
ii-  Correctly  Classify  by  Class  (Pcc  ). 

1.  Units:  probability, 

2.  Domain:  [0,l], 

3.  Weight:  0.526. 

4.  Value  Function:  exponential  fitting  {(0.5,0)(0.75,0.2)(l,l)). 

5.  Utility  Function: 

{(0.5,0)  (0.6,0.25)  (0. 7,0.5)  (0.9,0.75)  (l,l)}. 

6.  Cost 

.  Top  level  weight:  0.1. 

.  Subvalues: 

i.  Development 

ii.  Weight:  0.01. 

1 .  Money. 

a.  Units:  $M. 

b.  Domain:  [0,10], 

c.  Weight:  0.24. 

d.  Value  Function:  exponential  fitting 
{(o,i)  (5,0.3)  (l  0,0)}. 

e.  Utility  Function: 

{(0,1)  (2.5,0.75)  (5,0.5)  (7.5,0.25)  (l 0,0)}. 

2.  Time. 

a.  Units:  months. 

b.  Domain:  fo,  1 8] . 

c.  Weight:  0.2. 

d.  Value  Function:  exponential  fitting 
{(0,1)  (9,0.3)  (18,0)}. 

e.  Utility  Function: 

{(0,1)  (4.5,0.75)  (9,0.5)  (13.5,0.25)  (18, 0)}. 

3.  Expertise 
a.  Units:  categorical. 


27 


[320] 


b.  Domain:  {Technical  Training,  BS  in  Engineering, 
Graduate-level  Engineer,  Multi-location  subject 
matter  experts  (SME),  Single  Site  SME}. 

c.  Weight:  0.16. 

d.  Value  Function:  {(Technical  Training,  1),  (BS  in 
Engineering,  0.8),  (Graduate-level  Engineer,  0.4), 
(Multi-location  subject  matter  experts  (SME),  0.2), 
(Single  Site  SME,  0)}. 

e.  Utility  Function:  {(Technical  Training,  1),  (BS  in 
Engineering,  0.9),  (Graduate-level  Engineer,  0.8), 
(Multi-location  subject  matter  experts  (SME),  0.7), 
(Single  Site  SME,  0)}. 

4.  Risk 

a.  Units:  categorical. 

b.  Domain:  Low,  Medium,  and  High. 

c.  Weight:  0.4. 

d.  Value  Function:  exponential  fitting:  {(Low,  1), 

’  (Medium,  0.5),  (High,  0)}. 

e.  Utility  Function:  {(Low,  1),  (Medium,  0.8),  (High, 

0)}. 

iii.  Redeployment. 

iv.  Weight:  0.29. 

1 .  Money. 

a.  Units:  normalized  on  Global  Hawk. 

b.  Domain:  [o,l]. 

c.  Weight:  0.143. 

d.  Value  Function:  linear,  {(0,1 )  (l/3 ,0)  (l,0)}. 

e.  Utility  Function: 

{(0, l)(0. 15, 0,75)  (0.333, 0.5)  (0.9999, 0.25)  (l,0)}. 

2.  Time. 

a.  Units:  days. 

b.  Domain:  [0,90]. 

c.  Weight:  0.179. 

d.  Value  Function:  exponential  fitting 
{(0,1)  (45,0.2)  (90,0)}. 

e.  Utility  Function: 

{(0,1)  (15,0.75)(30,0.5)  (50,0.25)  (90, 0)}. 

3.  Expertise 

a.  Units:  categorical. 

b.  Domain:  {Technical  Training,  BS  in  Engineering, 
Graduate-level  Engineer,  Multi-location  subject 
matter  experts  (SME),  Single  Site  SME}. 

c.  Weight:  0.321. 

d.  Value  Function:  {(Technical  Training,  1),  (BS  in 
Engineering,  0.95),  (Graduate-level  Engineer,  0.9), 
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(Multi-location  subject  matter  experts  (SME),  0.5), 
(Single  Site  SME,  0)} , 

e.  Utility  Function:  {(Technical  Training,  1),  (BS  in 
Engineering,  0.95),  (Graduate-level  Engineer,  0.9), 
(Multi-location  subject  matter  experts  (SME),  0.8), 
(Single  Site  SME,  0)  }, 

4.  Risk 

a.  Units:  categorical. 

b.  Domain:  Low,  Medium,  and  High. 

c.  Weight:  0.357. 

d.  Value  Function:  exponential  fitting:  {(Low,  1), 
(Medium,  0.5),  (High,  0)}. 

e.  Utility  Function:  { (Low,  1),  (Medium,  0.2),  (High, 

0)}. 

v.  Use. 

vi.  Weight:  0.7. 

1.  Money. 

a. ’  Units:  normalized  on  Global  Hawk. 

b.  Domain:  [0,2], 

c.  Weight:  0.217. 

d.  Value  Function:  exponential  fitting 

{(0,1)  (1,0.2)  (2,0)}. 

e.  Utility  Function: 

{(0,1)  (0.5,0.75)  (1,0.5)  (1 .5,0.25)  (2,0)} 

2.  Time. 

a.  Units:  minutes. 

b.  Domain:  [0,5]. 

c.  Weight:  0.435. 

d.  Value  Function:  exponential  fitting 
{(0,1)  (2:  30,0.3)  (5,0)}. 

e.  Utility  Function: 

{(0,1)  (1,0.75)  (2,0.5)  (3,0.25)  (5,0)}. 

3.  Expertise 

a.  Units:  categorical. 

b.  Domain:  {Technical  Training,  BS  in  Engineering, 
Graduate-level  Engineer,  Multi-location  subject 
matter  experts  (SME),  Single  Site  SME}. 

c.  Weight:  0  348. 

d.  Value  Function:  {(Technical  Training,  1),  (BS  in 
Engineering,  0.4),  (Graduate-level  Engineer,  0.3), 
(Multi-location  subject  matter  experts  (SME),  0.1), 
(Single  Site  SME,  0)}. 

e.  Utility  Function:  {(Technical  Training,  1),  (BS  in 
Engineering,  0.2),  (Graduate-level  Engineer,  0.1), 
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(Multi-location  subject  matter  experts  (SME),  0.05), 
(Single  Site  SME,  0)}. 

7.  Self-Assessed  Accuracy. 

a.  Es-Pd 

i.  Units: 

ii.  Domain:  [0,1], 

iii.  Weight:  0.333. 

iv.  Value  Function:  exponential,  fitting  {(0,l)(0. 5,0,7  )(l,0)}. 

v.  Utility  Function:  {(0,l)  (0.25,0.75)  (0.5,0,5)  (0.75,0.25)  (l,0)}. 

b.  Es-Pid 

i.  Units: 

ii.  Domain:  [0,1], 

iii.  Weight:  0.333. 

iv.  Value  Function:  exponential  fitting  {(0,l)  (0.5, 0.7)  (l,o)}. 

v.  Utility  Function:  {(0, l) (0.25, 0.75) (0.5, 0.5)  (0.75, 0.25) (1,0)}. 

c.  Es-Pec 

i.  Units:  '  > 

ii.  Domain:  [0,1], 

iii.  Weight:  0.333. 

iv.  Value  Function:  exponential  fitting  {(0,l)  (0.5, 0.7)  (l,o)}. 

v.  Utility  Function:  {(0,1 )  (0.25,0.75)  (0.5.0.5)  (0.75,0.25)  (t,0)}. 
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List  of  Acronyms 


AFR.L 

Air  Force  Research  Laboratoiy 

AGRI 

Air-to-Ground  Radar  Imaging 

ATR 

Automatic  Target  Recognition 

CID 

Combat  Identification 

COMPASE  Center 

Comprehensive  ATR  Scientific  Evaluation  Center 

CS 

Classification  System 

DA 

Decision  Analysis 

ISR 

Intelligence/Surveillance/Reconnaissance 

MBT 

Main  Battle  Tank 

MSTAR 

Moving  and  Stationary  Target  Acquisition  and  Recognition 

ROC 

Receiver  Operating  Characteristic 

SAR 

synthetic  aperture  radar 

TGT 

target 
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Appendix  E.  ATR  CS  Hybrid  Utility  Detailed  Results 


Background.  An  application  of  decision  analysis  to  the  problem  of  selection  of 
an  Automatic  Target  Recognition  (ATR)  classification  system  (CS)  is  provided  in 
Chapter  V.  Detailed  infonnation  is  presented  below.  Simply  stated,  these  results  are  a 
series  of  group  screening  steps  investigating  the  significance  of  the  evaluation  measures. 

First  Group  Screening.  The  ATR  CS  hybrid  utility  infonnation  presented  in 
chapter  V  included  the  algorithm  results  involving  the  significant  variables.  The  detailed 

experimental  setup  is  shown  in  Table  59.  The  subscripts  for  the  exponential  constant 
variables  of  the  initial  group  screening,  pi ,  are: 

1 .  Robustness, 

2.  Overall  Detection  Performance, 

3.  Employment  Concept, 

4.  Declaration  Ability, 

5.  Classification  Ability, 

6.  Cost,  and 

7.  Self-Assessed  Accuracy. 

Explanation  of  the  variables  and  the  values  corresponding  to  the  settings  for  each  row  are 
presented  in  Chapter  V. 


Table  59.  Detailed  Experimental  Design  and  Results  for  ATR  Problem. 


Row 

Pattern 

A 

Pi 

A 

Pa 

Ps 

P6 

Pi 

w 

Profile 

Alternative 

U 

1 

-.1 

-.1 

-.1 

-.1 

-.1 

-.1 

-.1 

w~ 

CID 

ATR  33 

0.278 

2 

- o 

-.1 

-.1 

-.1 

-.1 

-.1 

-.1 

-.1 

w 

CID 

ATR  55 

0.302 

3 

- + 

-.1 

-.1 

-.1 

-.1 

-.1 

-.1 

-.1 

w~ 

CID 

ATR  89 

0.292 

4 

+.+ — +++. 

.1 

-.1 

.1 

-.1 

-.1 

-.1 

.1 

w+ 

ISR 

ATR  33 

0.535 

5 

+-+— +++0 

.1 

-.1 

.1 

-.1 

-.1 

-.1 

.1 

w+ 

ISR 

ATR  55 

0.505 
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Row 

Pattern 

A 

A 

A 

A 

A 

A 

A 

w 

Profile 

Alternative 

U 

6 

+.+ — ++++ 

.1 

-.1 

.1 

-.1 

-.1 

-.1 

.1 

w+ 

ISR 

ATR  89 

0.506 

7 

++.+ — ++. 

.1 

.1 

-.1 

.1 

-.1 

-.1 

-.1 

w+ 

ISR 

ATR33 

0.480 

8 

++-+— ++0 

.1 

.1 

-.1 

.1 

-.1 

-.1 

-.1 

w+ 

ISR 

ATR  55 

0.544 

9 

++.+ — +++ 

.1 

.1 

-.1 

.1 

-.1 

-.1 

-.1 

w+ 

ISR 

ATR  89 

0.546 

10 

-++-+ — +- 

-.1 

.1 

.1 

-.1 

.1 

-.1 

-.1 

w 

ISR 

ATR  33 

0.601 

11 

.++-+— +0 

-.1 

.1 

.1 

-.1 

.1 

-.1 

-.1 

w 

ISR 

ATR  55 

0.645 

12 

-++-+ — ++ 

-.1 

.1 

.1 

-.1 

.1 

-.1 

-.1 

w 

ISR 

ATR  89 

0.647 

13 

+-++-+ — 

.1 

-.1 

.1 

.1 

-.1 

.1 

-.1 

w” 

CID 

ATR  33 

0.574 

14 

+-++-+ — 0 

.1 

-.1 

.1 

.1 

-.1 

.1 

-.1 

w 

CID 

ATR  55 

0.622 

15 

+-++-+ — + 

.1 

-.1 

.1 

.1 

-.1 

.1 

-.1 

w” 

CID 

ATR  89 

0.627 

16 

++-++-+ — 

.1 

.1 

-.1 

.1 

.1 

-.1 

.1 

w~ 

CID 

ATR  33 

0.610 

17 

++-++-+— 0 

.1 

.1 

-.1 

.1 

.1 

-.1 

.1 

w 

CID 

ATR  55 

0.623 

18 

++-++-+-+ 

.1 

.1 

-.1 

.1 

.1 

-.1 

.1 

w~ 

CID 

ATR  89 

0.602 

19 

+++-++-+- 

.1 

.1 

.1 

-.1 

.1 

.1 

-.1 

w+ 

CID 

ATR  33 

0.772 

20 

+++-++-+-0 

.1 

.1 

.1 

-.1 

.1 

.1 

-.1 

w+ 

CID 

ATR  55 

0.806 

21 

+++-++-+-+ 

.1 

.1 

.1 

-.1 

.1 

.1 

-.1 

w+ 

CID 

ATR  89 

0.795 

22 

-+++-++-+- 

-.1 

.1 

.1 

.1 

-.1 

.1 

.1 

w~ 

ISR 

ATR  33 

0.512 

23 

-+++-++-+0 

-.1 

.1 

.1 

.1 

-.1 

.1 

.1 

w” 

ISR 

ATR  55 

0.565 

24 

-+++-++-++ 

-.1 

.1 

.1 

.1 

-.1 

.1 

.1 

w” 

ISR 

ATR  89 

0.583 

25 

-+++-++- 

-.1 

-.1 

.1 

.1 

.1 

-.1 

.1 

w+ 

CID 

ATR  33 

0.500 

26 

.-+++-++-0 

-.1 

-.1 

.1 

.1 

.1 

-.1 

.1 

w+ 

CID 

ATR  55 

0.545 

27 

--+++-++-+ 

-.1 

-.1 

.1 

.1 

.1 

-.1 

.1 

w+ 

CID 

ATR  89 

0.541 

28 

— +++-++- 

-.1 

-.1 

-.1 

.1 

.1 

.1 

-.1 

w+ 

ISR 

ATR  33 

0.381 

29 

— +++-++0 

-.1 

-.1 

-.1 

.1 

.1 

.1 

-.1 

w+ 

ISR 

ATR  55 

0.360 

30 

— +++-+++ 

-.1 

-.1 

-.1 

.1 

.1 

.1 

-.1 

w+ 

ISR 

ATR  89 

0.376 

31 

+ — +++-+- 

.1 

-.1 

-.1 

-.1 

.1 

.1 

.1 

w~ 

ISR 

ATR  33 

0.524 

32 

+ — +++-+0 

.1 

-.1 

-.1 

-.1 

.1 

.1 

.1 

w~ 

ISR 

ATR  55 

0.515 

33 

+ — +++-++ 

.1 

-.1 

-.1 

-.1 

.1 

.1 

.1 

w 

ISR 

ATR  89 

0.531 

34 

-+ — +++-- 

-.1 

.1 

-.1 

-.1 

-.1 

.1 

.1 

w+ 

CID 

ATR  33 

0.414 

35 

-+— +++-0 

-.1 

.1 

-.1 

-.1 

-.1 

.1 

.1 

w+ 

CID 

ATR  55 

0.433 

36 

-+ — +++-+ 

-.1 

.1 

-.1 

-.1 

-.1 

.1 

.1 

w+ 

CID 

ATR  89 

0.414 

37 

00000000- 

00 

00 

00 

00 

00 

00 

00 

w° 

CID 

ATR  33 

0.509 

38 

00000000-0 

00 

00 

oo 

00 

00 

00 

00 

w° 

CID 

ATR  55 

0.556 

39 

00000000-+ 

00 

00 

00 

oo 

00 

00 

00 

w° 

CID 

ATR  89 

0.525 

40 

00000000+- 

00 

00 

00 

00 

00 

00 

00 

w° 

ISR 

ATR  33 

0.497 

41 

00000000+0 

00 

00 

00 

00 

00 

00 

00 

w° 

ISR 

ATR  55 

0.531 
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Row 

Pattern 

A 

A 

Pi 

A 

A 

A 

A 

w 

Profile 

Alternative 

U 

42 

00000000++ 

00 

00 

00 

oo 

00 

00 

00 

w° 

ISR 

ATR  89 

0.497 

Second  Group  Screening.  The  group  screening  experimental  design  for  the 
robustness  group  is  presented  in  Table  60. 


Table  60.  Robustness  Screening  Experimental  Design  and  Response  for  ATR  Problem. 


Row 

Pattern 

A 

A 

A 

w 

Profile 

Alternative 

U 

1 

— ++- 

-.1 

-.1 

-.1 

w+ 

ISR 

ATR  33 

0.0289 

2 

+++ — 

.1 

.1 

.1 

w~ 

CID 

ATR  33 

0.949 

3 

-++ — 

-.1 

.1 

.1 

w~ 

CID 

ATR  33 

0.577 

4 

+-++- 

.1 

-.1 

-.1 

w+ 

ISR 

ATR  33 

0.745 

5 

-.1 

.1 

-.1 

w~ 

ISR 

ATR  33 

0.577 

6 

+-++-- 

.1 

-.1 

.1 

w+ 

CID 

ATR  33 

0.745 

7 

-++-0 

-.1 

-.1 

.1 

w+ 

CID 

ATR  55 

0.286 

8 

++-+0 

.1 

.1 

-.1 

w~ 

ISR 

ATR  55 

0.996 

9 

-+-+0 

-.1 

-.1 

.1 

w~ 

ISR 

ATR  55 

0.331 

10 

++-+-0 

.1 

.1 

-.1 

w+ 

CID 

ATR  55 

0.995 

11 

-+-+-0 

-.1 

.1 

-.1 

w+ 

CID 

ATR  55 

0.534 

12 

+-+-+0 

.1 

-.1 

.1 

w~ 

ISR 

ATR  55 

0.708 

13 

-+++++ 

-.1 

.1 

.1 

w+ 

ISR 

ATR  89 

0.526 

14 

+ — + 

.1 

-.1 

-.1 

w~ 

CID 

ATR  89 

0.701 

15 

- + 

-.1 

-.1 

-.1 

w~ 

CID 

ATR  89 

0.324 

16 

++++++ 

.1 

.1 

.1 

w+ 

ISR 

ATR  89 

0.987 

17 

- + 

-.1 

-.1 

-.1 

w~ 

CID 

ATR  89 

0.324 

18 

++++++ 

.1 

.1 

.1 

w+ 

ISR 

ATR  89 

0.987 

19 

000011 

OO 

00 

00 

w° 

CID 

ATR  33 

0.618 

20 

000012 

00 

00 

00 

w° 

CID 

ATR  55 

0.671 

21 

000013 

00 

00 

00 

w° 

CID 

ATR  89 

0.584 

22 

000021 

00 

00 

00 

w° 

ISR 

ATR  33 

0.618 

23 

000022 

oo 

00 

00 

w° 

ISR 

ATR  55 

0.671 

24 

000023 

00 

00 

00 

w° 

ISR 

ATR  89 

0.584 

326 


Third  Group  Screening.  The  group  screening  experimental  design  for  the  cost 
group  is  presented  in  Table  61. 


Table  61.  Cost  Screening  Design  and  Response  for  ATR  Problem. 
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Row 

Pattern 

A 

Pi 

A 

A 

A 

A 

A 

A 

A 

A  o 

Ai 

w 

ATR 

0 

36 
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.1 

-.1 
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.1 

,i 

-,i 

.1 

.1 

.1 

.1 

w 

89 
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.1 

-.1 

.1 

,i 
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.1 

.1 
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33 
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38 
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.1 

.1 
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.1 

.1 
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,i 
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w 

55 
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39 
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.1 
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.1 

.1 
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-.1 

.1 
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.1 
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89 
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.1 
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.1 

w+ 
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.1 

.1 
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.1 
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.1 

w 
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.1 
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.1 

-.1 

-.1 

.1 

w+ 

55 

0.475 
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.1 
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.1 
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,i 
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.1 

.1 

w+ 

89 

0.837 

46 
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.1 

.1 

-,i 

,i 

.1 

.1 

.1 

-.1 

w+ 

33 

0.778 

47 
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.1 
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.1 

.1 

.1 

-,i 
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.1 
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-.1 

.1 

w 

55 

0.438 

48 
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-.1 

.1 

-.1 

.1 

.1 

,i 

,i 

-.1 

.1 

-.1 

.1 

w 

89 
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49 

00000000000- 

00 

00 

00 

00 

00 

00 

00 

00 

00 

00 

00 

w° 

33 

0.719 

50 

000000000000 

00 

00 

00 

00 

00 

00 

00 

00 

00 

00 

00 

w° 

55 

0.500 

51 

00000000000+ 

00 

00 

00 

00 

00 

00 

00 

00 

00 

00 

00 

w° 

89 

0.670 

Fourth  Group  Screening.  The  group  screening  experimental  design  for  the 
detection  performance  group  is  presented  in  Table  62. 


Table  62.  Detection  Perfonnance  Screening  Experimental  Design  and  Response  for  ATR 

Problem. 


Row 

Pattern 

A 

A 

w 

Profile 

ATR 

U 

1 

...+. 

-.1 

-.1 

w 

ISR 

ATR  33 

0.0850 

2 

+++- 

.1 

.1 

w+ 

CID 

ATR  33 

0.680 

3 

-.1 

.1 

w+ 

CID 

ATR  33 

0.317 

4 

.1 

-.1 

w 

ISR 

ATR  33 

0.0850 

5 

-+— 

-.1 

.1 

w 

CID 

ATR  33 

0.260 

6 

+-++- 

.1 

-.1 

w+ 

ISR 

ATR  33 

0.104 

7 

-++0 

-.1 

-.1 

w+ 

ISR 

ATR  55 

0.0133 

8 

++--0 

.1 

.1 

w 

CID 

ATR  55 

0.919 

9 

-+-0 

-.1 

-.1 

w+ 

CID 

ATR  55 

0.331 

10 

++-+0 

.1 

.1 

w 

ISR 

ATR  55 

0.996 

11 

-+-+0 

-.1 

.1 

w 

ISR 

ATR  55 

0.522 

12 

+-+-0 

.1 

-.1 

w+ 

CID 

ATR  55 

0.373 

13 

-++++ 

-.1 

.1 

w+ 

ISR 

ATR  89 

0.626 

14 

+— + 

.1 

-.1 

w 

CID 

ATR  89 

0.486 

15 

....+ 

-.1 

-.1 

w 

CID 

ATR  89 

0.383 
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A 

Pi 

w 

Profile 

ATR 

U 

16 
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.1 

.1 

w+ 

ISR 

ATR  89 

0.991 

17 

....+ 

-.1 

-.1 

w 

CID 

ATR  89 

0.383 

18 

+++++ 

.1 

.1 

w+ 

ISR 

ATR  89 

0.991 

19 

000- 

00 

00 

w° 

CID 

ATR  33 

0.204 

20 

000-0 

00 

00 

w° 

CID 

ATR  55 

0.570 

21 

000-+ 

00 

00 

w° 

CID 

ATR  89 

0.464 

22 

000+- 

00 

00 

w° 

ISR 

ATR  33 

0.468 

23 

000+0 

00 

00 

w° 

ISR 

ATR  55 

0.562 

24 

000++ 

00 

00 

w° 

ISR 

ATR  89 

0.480 

Fifth  Group  Screening.  The  group  screening  experimental  design  for  the 
classification  ability  group  is  presented  in  Table  63. 


Table  63.  Classification  Ability  Screening  Design  and  Response  for  ATR  Problem. 


Row 

Pattern 

A 

Pi 

w 

Profile 

ATR 

U 

1 

— +- 

-.1 

-.1 

w 

ISR 

ATR  33 

0.00106 

2 
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.1 

.1 

w+ 
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-.1 

.1 

w+ 
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0.552 

14 
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15 

— -+ 
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w 
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0.0436 
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.1 

.1 

w+ 
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19 

00011 

00 

00 

w° 

CID 

ATR  33 

0.811 

20 

00012 

00 

00 

w° 

CID 

ATR  55 

0.769 

21 

00013 

00 

00 

w° 

CID 

ATR  89 

0.655 

22 

00021 

00 

00 

w° 

ISR 

ATR  33 

0.293 
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23 

00022 

00 

00 

W° 

ISR 

ATR55 

0.557 

24 

00023 

00 

00 

W° 

ISR 

ATR  89 

0.374 

Hybrid  Value-Utility  Algorithm.  The  results  from  applying  the  algorithm  to  all 
evaluation  measures  for  each  mission  profile  are  presented  here. 


Table  64.  CID  Profile  Value,  Utility,  and  Hybrid  Utility  Results,  All  Evaluation 

Measures. 


Notes 

ATR  33 

ATR  55 

ATR  89 

Goal 

Utility  Model 

0.572 

0.507 

0.518 

Iteration 

0 

Value  Model 

0.509 

0.556 

0.525 

1 

Employment  Concept 

0.464 

0.511 

0.480 

2 

Detection  Robustness 

0.455 

0.471 

0.442 

3 

Probability  of  False  Alarm 

0.529 

0.500 

0.512 

4 

Classification  by  Class 

0.536 

0.509 

0.524 

5 

Classification  by  Type 

0.538 

0.511 

0.528 

6 

Identification  Robustness 

0.529 

0.505 

0.509 

7 

False  Alarm  Rate 

0.569 

0.506 

0.510 

8 

Declaration  Ability 

0.569 

0.506 

0.510 

9 

Cost  Use  Time 

0.569 

0.506 

0.513 

10 

Classification  Robustness 

0.569 

0.506 

0.513 

11 

Cost  Redeployment  Time 

0.570 

0.507 

0.513 

12 

Cost  Use  Expertise 

0.570 

0.507 

0.513 

13 

Cost  Redeployment  Risk 

0.570 

0.504 

0.513 

14 

Cost  Redeployment 
Expertise 

0.570 

0.506 

0.516 

15 

Cost  Use  Money 

0.570 

0.507 

0.516 

16 

Cost  Development 
Expertise 

0.570 

0.507 

0.516 

17 

Redeployment  Money 

0.572 

0.507 

0.518 

18 

Cost  Development  Money 

0.572 

0.507 

0.518 

19 

Cost  Development  Risk 

0.572 

0.507 

0.518 

20 

Cost  Development  Time 

0.572 

0.507 

0.518 

21 

Es  Pd 

0.572 

0.507 

0.518 

22 

Es  Pid 

0.572 

0.507 

0.518 

23 

Es  Pec 

0.572 

0.507 

0.518 
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Table  65.  ISR  Profile  Value,  Utility,  and  Hybrid  Utility  Results,  All  Evaluation 

Measures. 


Notes 

ATR33 

ATR55 

ATR  89 

Goal 

Utility  Model 

0.415 

0.455 

0.439 

Iteration 

0 

Value  Model 

0.497 

0.531 

0.500 

1 

Employment  Concept 

0.452 

0.486 

0.452 

2 

Detection  Robustness 

0.443 

0.446 

0.414 

3 

Probability  of  False  Alarm 

0.371 

0.415 

0.383 

4 

Classification  by  Class 

0.392 

0.430 

0.403 

5 

Classification  by  Type 

0.405 

0.435 

0.413 

6 

Identification  Robustness 

0.395 

0.429 

0.394 

7 

False  Alarm  Rate 

0.412 

0.455 

0.432 

8 

Declaration  Ability 

0.412 

0.455 

0.432 

9 

Cost  Use  Time 

0.412 

0.455 

0.434 

10 

Classification  Robustness 

0.412 

0.455 

0.434 

11 

Cost  Redeployment  Time 

0.413 

0.455 

0.434 

12 

Cost  Use  Expertise 

0.413 

0.455 

0.434 

13 

Cost  Redeployment  Risk 

0.413 

0.452 

0.434 

14 

Cost  Redeployment 
Expertise 

0.413 

0.455 

0.437 

15 

Cost  Use  Money 

0.413 

0.455 

0.437 

16 

Cost  Development 
Expertise 

0.413 

0.455 

0.437 

17 

Redeployment  Money 

0.415 

0.455 

0.439 

18 

Cost  Development  Money 

0.415 

0.455 

0.439 

19 

Cost  Development  Risk 

0.415 

0.455 

0.439 

20 

Cost  Development  Time 

0.415 

0.455 

0.439 

21 

Es  Pd 

0.415 

0.455 

0.439 

22 

Es  Pid 

0.415 

0.455 

0.439 

23 

Es  Pec 

0.415 

0.455 

0.439 
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